THE CHAPTER 3: METHODOLOGY

http://www.edu.plymouth.ac.uk/resined/resedhme.htm

A Mixed Method Design

Date Published: 13th July 2007

Author: Olivia Hunt

RSS

ASK ABOUT THIS ARTICLE

This article is devoted to a mixed method research design. In the article special attention is paid to quantitative and qualitative approach for the design of a research as essential components of a mixed research paradigm. There are main research paradigms in education and science. They are: quantitative research, qualitative research and mixed research.

A mixed research design is a general type of research that includes quantitative and qualitative research data, techniques and methods. All these paradigm characteristics are mixed in one case study. This method design involves research that uses mixed data (numbers and text) and additional means (statistics and text analysis). A mixed method uses both deductive and inductive scientific method, has multiple forms of data collecting and produces eclectic and pragmatic reports.

Two main types of a mixed method are: mixed method and mixed model research. A mixed research method is a research in which you use quantitative data for one stage of a research study and qualitative data for a second stage of a research. A mixed model design is a research in which you use both quantitative and qualitative data in one or two stages of the research process. The mixing of quantitative and qualitative approaches happens in every stage of a research.

In a research it is important to use a mixed research method for the conducting of a detailed research. The advantages of a mixed research are:
1) The strength of the research;
2) Use of multiple methods in a research helps to research a process or a problem from all sides;

3) Usage of different approaches helps to focus on a single process and confirms the data accuracy. A mixed research complements a result from one type of research with another one. This research does not miss any available data.
A quantitative component of a mixed research assumes the usage of deductive scientific method while qualitative component assumes inductive scientific method. Moreover, a quantitative approach collects quantitative data based on exact measurement applying structured as well as validated information collection. For instance, rating scales, closed-ended items and responses. This approach produces statistical report with correlations.

A qualitative component uses qualitative information. For instance, interview, field notes, open-ended questions etc. This approach considers a researcher to be the major means of information collection. At the end of a research this approach supposes a narrative report with context description, quotations taken from research material.

It is important to stress that there are many ways of research performing. Quantitative and qualitative methods have their advantages and disadvantages in a research. However, you may summarize the advantages of both methods and have accurate information on implementation, findings and conclusions of your research project. Qualitative and quantitative research methods have different strengths, weaknesses and requirements that affect researcher’s project accuracy. The aim of a mixed method design is to summarize positive aspects of two approaches and produce a highly accurate data.

When you use several methods in your research process, then you can use the strength of every type of information collection and minimize the weak points of every of both approaches. A mixed method approach of gathering and evaluation can increase the validity and accuracy of the information. The article briefly analyzes a mixed method research design including the major components: quantitative and qualitative approach for the design of a research. The article proves the effectiveness of a mixed method design.

The article was produced by the writer of Essay-Paper.net. Olivia Hunt is a 4-years experienced freelance writer of Essay Writing Service. Contact her to get information about essay writing service and online writing at our website.

This article is free for republishing
Source: http://www.articlealley.com/article_185975_22.html

Questionnaires in Education Research

Interviews in Education Research

Materials originally prepared by Professor Andrew Hannan.

Now led by Helen Knowler.

CONTENTS

A. INTRODUCTION

B. HOW TO USE INTERVIEWS IN EDUCATION RESEARCH

1. Why should I use interviews rather than some other device for data collection?

2. Whom should I interview?

3. How will I get access?

4. What questions should I ask?

5. How should I phrase my questions?

6. What techniques should I use get the best quality answers?

7. How do I decide on what form of interview to use?

8. What about group interviews or 'focus group' discussions?

9. How will I record the interview?

10. How do I analyse the data?

11. How do I write up the findings?

12. What should I avoid?

C. EXAMPLES

D. TASKS

E. FURTHER READING

A. Introduction

Interviews can be used to collect facts, eg information about people's place of work, age, etc., but such questions are usually no more than opening items which precede the main substance. The bulk of interview questions seek to elicit information about attitudes and opinions, perspectives and meanings, the very stuff of much of both psychology and sociology. Interviews are also in common use as a means of selection - for entry to school or college, getting a job or obtaining promotion. They are widely used because they are a powerful means of both obtaining information and gaining insights. We use them because they give us an idea of 'what makes people tick', of the personality and the motivations of the interviewee.

Interviews are available in a range of styles, some of which are pre-packed and mass marketed so they can be more or less picked off the shelf. If you have ever been stopped in the high street to be quizzed about your use of toiletries, you'll know what a closed-ended, structured interview feels like on the receiving end. Social scientists make similar use of tightly controlled pre-set interviews which have been piloted on sample groups to test their efficiency and accuracy before being tried out on larger populations.

These structured interviews in their simplest form are sometimes little more than oral questionnaires - used instead of the written form in order to obtain a higher response rate or with respondents, especially children, who might not be literate or capable of correctly completing a complex questionnaire.

At the opposite extreme in interview design are completely unstructured conversations between researcher and respondent, where the latter has as much influence over the course of the interview as the former.

There is a half-way house, where the researcher designs a set of key questions to be raised before the interview takes place, but builds in considerable flexibility about how and when these issues are raised and allows for a considerable amount of additional topics to be built in in response to the dynamics of conversational exchange. These are known as semi-structured interviews. They are the form most often used in education research.

B. How to use interviews in education research

One way to proceed would be to ask yourself the following series of questions:

1. Why should I use interviews rather than some other device for data collection?

Try and generate your own lists of what interviews are good at (strengths) and what they are not good at (weaknesses). Compare your lists to those I've compiled by clicking here. Carefully consider the nature of your research question and the resources you have available - would it be better to use other data collection techniques, eg a questionnaire or participant observation? Could you use a variety of methods to triangulate, ie to elicit information by means of a number of different devices so as to be more confident of its accuracy? Interviews are a very useful means of inquiry, but they do not fit all circumstances. For a critique from a postmodernist perspective, see Scheurich (1995) and Case Study 3 from the RESINED component on Education Research in the Postmodern.

Beware of the fact that interviews themselves are contrived, artificial situations and that interviewees often respond to them in a manner that reflects this. Thus, they may describe what they do in terms that they think you will recognise, making full play of theoretical concepts intended to impress you, or describe what they think they should be doing rather than what they actually do. Teachers interviewed about their pedagogic styles often respond with descriptions drawn from the 'educationist context' that are very different from the manner in which they may be observed to operate in the 'practitioner context' of the classroom.

2. Whom should I interview?

As in all surveys (using either questionnaire or interview as the means of data collection), you should attempt to obtain as big a response as possible. However, as your survey is using a research instrument that is time-intensive, you will probably be considering a limited number of cases, chosen for particular interest. Your cases are more likely, for example, to be individual teachers than whole school staffs. You'll be looking to pick out those who can provide the best insights, who represent the full range of experience and opinion, who can be said to be typical if you claim representativeness or to be illustrative of certain types if you want to explore theoretical models. You may decide to interview just those you consider to be the most important informants, the most powerful or, perhaps, the least. You may have little choice but to use an 'opportunity sample', ie the people who happened to be available. Whatever choice of sample you make you need to justify it, ie to make a case to the reader who examines your results that he/she has good grounds for taking your findings seriously in terms of their representativeness.

Peter Woods discusses sampling in qualitative research in education in his RESINED component on Qualitative Research:

Where qualitative research is seeking to generalise about general issues, representative or 'naturalistic' sampling is desirable. This covers places, times and persons. Thus, if we were studying teachers' or pupils' perspectives, or the culture of a group, we would need to consider them in different settings, since behaviour can differ markedly in different situations - for example, the formal circumstances of a teacher's classroom or office, the staffroom, different classrooms, the informal ambience of a pub, and the personal stronghold of the teacher's home. The same point applies to time. Weekly and yearly cycles, for example, are critical in schools. If our research sampled at just the beginnings and/or ends of terms, weeks or days, we would end up with a distorted study if we were to claim our results applied more generally. Again, if we are seeking to represent a group in our findings (the 'English Department', the 'Year 10 Girls'), we should ensure that we have sampled across that group according to some appropriate criteria, such as age, gender, ethnicity, experience.

Representative sampling cannot always be achieved in qualitative research because of a) the initially largely exploratory nature of the research; b) problems of negotiating access; c) the sheer weight of work and problems of gathering and processing data using only one set of eyes and ears. Often, one has to make do with an opportunity sample in those areas where access is offered; or a snowball sample, where the sample is developed through personal contact and recommendation as the research proceeds. In these cases, the basis of the sampling must be made clear and no inappropriate generalising claims made for the findings.

For a discussion of survey research from a largely quantitative perspective, visit http://www.socialresearchmethods.net/kb/survtype.htm

For advice on using personal interviews in survey research of this kind, visit: http://www.socialresearchmethods.net/kb/intrview.htm

For information about sampling in large-scale surveys, visit: http://www.socialresearchmethods.net/kb/sampling.htm

All these links take you to The Research Methods Knowledge Base, 2nd Edition, Trochim, William M. Internet WWW page, at URL: http://www.socialresearchmethods.net/kb/ (version current as of September 24, 2004).

3. How will I get access?

Generally speaking, people are quite flattered by attention and this sometimes overcomes any inhibitions so that matters quite secret are paraded before the interviewer with seeming abandon. At other times, particularly where an issue has been the subject of recent press attention, the shutters go up and there is no way in. All you can do in such circumstances is to give up and try something else likely to be more productive - you haven't the time to spend on lengthy negotiation. Crucial to the business of initially gaining access is the whole matter of assurances of confidentiality, anonymity, etc, as set out in your ethics protocol (see http://www.edu.plymouth.ac.uk/resined/beginning/begresed.htm#Ethics). It is advisable to produce an easily readable version of your ethics protocol that you can send in advance or provide for your informants before you begin the interview. They should feel happier telling you all they know if they are confident that you won't use the information in any way that will harm them, that you will respect confidentiality, that you will seek their approval before using anything that might reveal their identity, that you will provide them with a transcript or a copy of your notes, that you will show them how you intend to make use of what they've said in the report you write (perhaps giving them the power to respond or withdraw their statements), that you will stop the recording at any point if requested, etc. It's up to you which of these or other promises you might make, but it's best to have them up front so that the nature of the contract between you as information seeker and provider is explicit from the beginning. For examples of ethics protocols visit: http://www.edu.plymouth.ac.uk/resined/beginning/four%20examples%20of%20ethics%20protocols.doc

It is not at all easy to get access to the perspectives of subordinates, eg if you are a headteacher wanting to know if an appraisal scheme is working well from the viewpoint of heads of department, a teacher wishing to interview pupils about their preferred learning styles or a lecturer who wants to know what your students think about life in college. As a powerful actor, the responses of others to your questions will inevitably reflect their views of you, what they think you want them to say and what they think might be the consequences of their answers. Interviews reveal identities and the lack of anonymity limits openness. Questionnaires, however, can be anonymous and might be preferable if you are trying to get at the truth, eg if you want to evaluate your teaching by asking learners to rate your effectiveness. To some extent you can overcome some of the limitations of interviews in such circumstances if you make cast-iron promises on confidentiality, if you 'triangulate' by collecting information about the same issues using other devices (eg through observation, questionnaire survey and interview) or, more radically, if you get the informants to interview one another on your behalf. Andrew Pollard (1985, p57) did the latter in his own PhD research:

My concern to understand children's perspectives meant that I had to find a way of collecting data which minimised the possible distorting effect of being seen as a teacher. ... the key procedure which I adopted, of working with a team of child interviewers, requires a brief description here. The crucial initiative was to start a dinner-time club for fourth-year children concerned with 'finding out what children think about school'. The children who came to the club regularly called it MID, the Moorside Investigation Department, and it seemed to capture their imagination. Club members invited other children to be interviewed, initiated discussions and emphasised the 'top secret' and confidential nature of the activity. After a period in which confidence and trust developed I became more involved and worked alongside the child interviewers. Interviews were recorded on cassettes and later transcribed. The children also discussed many elements of my analysis with me as it emerged.

4. What questions should I ask?

When thinking about the questions to be posed in an interview, you should have in mind the overall research questions with which you are concerned.

The fundamental question that must then be asked is, what are you trying to find out? Every interview must have a purpose, ie it must draw from some underlying hypotheses about what are the important facts or opinions and even make some predictions about which facts may be relevant in explaining the opinions expressed.

Write your own rationale, in terms of statements like, ‘I need to know whether or not senior members of staff are more likely to support the moves to introduce appraisal and what reasons they have for the positions they express. I need to find out why junior staff seem opposed, ie are they misinformed about the nature of the reforms or are they protecting weaker colleagues from what is seen as scapegoating in an under-funded profession of whom too much is demanded with too little support?’

This can be developed so as to produce a justification for every question used, eg ‘I asked this so as to probe the extent to which those of various positions in the hierarchy valued staff consensus and the feeling of shared purpose, with the intention of seeing whether those who were strongly committed to such views were also more or less opposed to staff appraisal’. If you can’t come up with a good rationale, drop the question.

5. How should I phrase my questions?

a) Bias versus rapport

On the one hand you will want to avoid bias, to be careful not to ask questions in such a way that you lead respondents into providing confirmation of your own views rather than eliciting theirs. You will need to be conscious of the interviewee's probable wish to please you, to defer to your expertise, to seek your (nodded) approval. On the other hand, you will want to get the best possible responses by creating a positive relationship with the person with whom you are talking, building up some empathy and developing rapport. This is a fine line to tread! Ideally, interviewers should be monitored in some way to ensure that they do not overstep the mark. If you are part of a team, this can be done by attending one another's interviews or listening to recordings (or watching video recordings). You should at least give the reader the chance to assess possible influences of this kind by providing the questions posed as well as the answers they obtained when presenting your findings.

b) Thought-provoking questions

At least some of your questions should be designed so as to promote thought. Harold Silver used the following questions in the Innovations in Teaching and Learning in Higher Education project at the end of interviews to get the informants to relax a little, step back from the immediate and the detailed and think about the longer term:

(To a lecturer in an interview about introducing new methods of teaching in HE )

If you were enjoying your retirement on a desert island, sipping a martini, looking back on your time here, would you say you were an innovator and if so, why?

(To a senior manager)

If you were invited to speak at an international conference about innovation in teaching and learning in this university would you attend and if so, what would you speak about?

c) Difficult topics

Sometimes you will need to use what Wragg (1978) calls 'protective techniques' to handle difficult topics. He gives a number of examples, including the following (p 19):

'Guess who' technique - A researcher investigates under-age smoking. He knows that few children will admit to it, so he says, 'Guess who this describes. This person often smokes secretly. Without telling me the names look through this class list and tell me how many people it might be'. If a whole class come up with a number like 12, 13 or 14 the investigator would have a rough idea of how many were involved without anyone having given away his friends.

Critical event - When people talk about some issues their language is often vague. For example, teachers often talk about classes being 'busy' or someone 'having a firm grip'. A clearer picture of what this means can emerge if the person is asked to describe a few 'critical events', ie things which happen in a lesson which indicated that the class was 'busy' or that a certain teacher had 'a firm grip'.

6. What techniques should I use get the best quality answers?

Peter Woods has the following advice to give those undertaking unstructured interviews in his RESINED component on Qualitative Research, much of which can also be applied by those making use of semi-structured interviews:

Attention will be paid to where the interview is held, arrangement of seating, how the researcher dresses, manner of approach, all in the interests of equality. There might be a certain amount of pleasant chat before getting into explaining what the research is about. If rapport has been established, there should not be a difficulty in getting people to talk. The problem, rather, might be that they talk too inconsequentially, or off the subject, or vaguely. There are a number of techniques researchers use in the natural course of the conversation to aid clarity, depth and validity. Here are some:

check on apparent contradictions, non sequiturs, imbalance, implausibility, exaggerations, or inconsistencies ('Yes, but didn't you say a moment ago…?' 'How can that be so if…?' 'Is it really?' 'Does it necessarily follow that…?' 'Why?' 'Why not?' 'What was the point of that?');

search for opinions ('What do you think of that?' 'Do you believe that?');

ask for clarification ('What do you mean by…?' 'Can you say a little more about…?' 'In what way?' 'Can you give me some examples?');

ask for explanations, pose alternatives ('Couldn't one also say…?');

seek comparisons ('How does that relate to…?' 'Some others have said that…');

pursue the logic of an argument ('Does it follow, then, that…?' 'Presumably,…?');

ask for further information ('What about…?' 'Does that apply to…?);

aim for comprehensiveness ('Have you any other…?' 'Do you all feel like that?' 'Have you anything more to say on that?');

put things in a different way ('Would it be fair to say that…?' 'Do you mean…?' 'In other words…?');

express incredulity or astonishment ('In the fourth year?' 'I don't believe it!' 'Really??');

summarise occasionally and ask for corroboration ('So…?' 'What you're saying is…?' 'Would it be correct to say…?');

ask hypothetical questions ('Yes, but what if…?' 'Supposing…?');

play devil's advocate ('An opposing argument might run…' 'What would you say to the criticism that…?).

The researcher engages in 'active' listening, which shows the interviewee that close attention is being paid to what they say; and also tries to keep the interviewee focused on the subject, as unobtrusively as possible. Something of the researcher's self - perhaps involving some similar or contrasting experiences to those of the interviewee - is also put into the interaction in the interests of sustaining rapport and encouraging more discussion. In this sense, the unstructured interview is a process of constructing reality to which both parties contribute.

Using other forms of data to prompt discussion in an interview (eg information taken from observation, a questionnaire or a diary) can also be very useful. For information about this method, known as 'Interview Plus', see http://www.jisc.ac.uk/media/documents/lex_method_final.doc

7. How do I decide on what form of interview to use?

Peter Woods, again in his RESINED component Qualitative Research, discusses the three major varieties of individual interview:

A great deal of qualitative material comes from talking with people whether it be through formal interviews or casual conversations. If interviews are going to tap into the depths of reality of the situation and discover subjects' meanings and understandings, it is essential for the researcher:

to develop empathy with interviewees and win their confidence;

to be unobtrusive, in order not to impose one's own influence on the interviewee.

The best technique for this is the unstructured interview. Here, the researcher has some general ideas about the topics of the interview, and may have an aide memoire of points that might arise in discussion for use as prompts, if necessary. But the hope is that those points will come up in the natural course of the discussion as the interviewee talks. Care is needed, therefore, to avoid leading questions or suggesting outcomes, and skill is called for in discovering what the interviewee really thinks. The researcher aims to appear natural, not someone with a special role, but one who engages with interviewees on a person-to-person basis. ....

As with observation, it may be that the researcher begins with a more focused study and wishes to know certain things. In these cases a structured interview might be more appropriate. Here the researcher decides the structure of the interview and sets out with predetermined questions. As with systematic observation, this is less naturalistic. Within the spaces, the same techniques as above might apply, but there is clearly not as much scope for the interviewee to generate the agenda. For this reason, some researchers use semi-structured interviews - interviews which have some pre-set questions, but allow more scope for open-ended answers.

Both kinds of interview might be used in the same research. For example, the initial stage of a project might be exploratory and expansive. But once certain issues have been identified, the researcher might use more focused interviews. They are still grounded in the reality of the situation.

Fundamentally, then, the degree to which the interview is structured, ie the extent to which the content and order of questions are pre-determined, depends on the nature of the research itself and the overall approach to be adopted.

8. What about group interviews or 'focus group' discussions?

Click here to see notes on using focus groups in education research prepared by Professor Rosemary Deem, then of the Department of Educational Research of Lancaster University and now of the University of Bristol (used with permission). I also recommend that you should read Wilson (1997) and, in particular, pages 72-82 in A Student's Guide to Methodology, by Clough & Nutbrown (2002).

For further advice visit: Conducting Focus Groups.

An example of an agenda for a focus group discussion is given in C. Examples, below.

9. How will I record the interview?

Remember that the more you succeed in recording every possible detail of what took place, the more data you will have to analyse. It is all too easy to be swamped by too much information and there is a danger that you will miss seeing the wood for all the trees. Audio-recordings are easy to make now that we have mini tape recorders and sensitive microphones, but transcription can be very time-consuming (often taking four times as long as the interview itself). Video-recordings have the advantage of capturing facial expressions, etc, but it's difficult to get both interviewer and interviewee and the problems don't end there as the researcher still has to find a way of coding visual data (not an easy task!). It should also be taken into account that those you wish to interview may be more reluctant to take part or to reveal their true thoughts the more thorough the recording technique. Interviewees frequently say much more once the tape recorder has been switched off, or give an entirely different view when having a chat over a cup of tea in the staffroom than they do when confronted with a microphone.

On the other hand, relying on what you have remembered from unrecorded conversations is likely to be unreliable to some extent, although the sooner you can write down what you've heard the better (distinguishing between verbatim and summarised statements wherever possible). You may not get the precise wording right, but you may get insights in this manner that you would not obtain in more formal contexts. However, given the circumstances of the data collection, you may not be able to attribute the source of such information or you may have to get your informants to approve your use of such material.

Many experienced interviewers rely on taking notes whilst the interviews are taking place. Sometimes this is done with the aid of pre-determined categories for possible responses that the interviewer merely has to tick, but to which can be added additional observations. In my own research, I have developed a very effective method of note-taking that relies on a shorthand that only I can decipher, which includes both verbatim quotations of the more 'juicy' statements and paraphrased summaries of the positions taken by the interviewee. I take these notes whether or not I am also tape-recording the conversation. That way, I still have some record if the machine fails (not switched on, batteries flat, volume too low, tape put in the wrong way around, etc). If I have both my notes and the tape I can save myself some time when transcribing by listening for the bits I already know to be important. I have found that interviewees seem to make allowances for my note-taking, giving me time to jot things down and generally pacing themselves to my benefit.

You can, of course, pay for or persuade someone else to do your transcribing for you. However, this isn't as simple as it seems since an outsider is unlikely to be as familiar as you with the context and will not recognise the jargon used. Such transcripts often have to be extensively corrected by the researcher re-playing the tape to plug up on gaps and pick up on mistakes. Full transcriptions are also often very long, including much information that isn't necessary.

The biggest advantage of doing the transcription yourself is that you know what's being discussed. It is possible for you to replay tapes making notes as you go summarising the points made and then pausing the tape, rewinding and writing out verbatim those bits that are of particular interest. A transcription machine, involving pedal controls to free the hands for typing, etc., helps a great deal. You also get to know the data better the more you listen, constructing categories as you go and analysing what has been said as part of the process of testing and generating hypotheses.

Click here to read Tinkering with Transcriptions by Phil Bayliss, a very useful and interesting discussion about transcribing.

In any case, I strongly recommend that you pilot your schedule and test your method of recording before you undertake your project itself. It is also a good idea to undertake some preliminary analysis of the data collected in a pilot so that you know you are getting the sort of information you need.

10. How do I analyse the data?

Excellent advice on this is given by Peter Woods in his RESINED component Qualitative Research in section 3 on QUALITATIVE ANALYSIS. This includes several examples of interpretative analysis, based on actual interviews.

There are lots of software packages which facilitate content analysis of interview responses (see http://www.soc.surrey.ac.uk/caqdas/). Weft QDA is a free qualitative analysis software application - 'an easy-to-use tool to assist in the analysis of textual data such as interview transcripts, written texts and fieldnotes', which can be downloaded from http://www.pressure.to/qda/. N6 and NVivo are provided over the University of Plymouth server to all networked PCs. If you can’t find these via Start > All Programs > Software M to O > N > N6 or NVivo 2 try installing via Start -> Settings -> Control Panel -> Run Advertised Programs. Copies of the CD-ROM for home installation can also be obtained via the library (Media Services counter at Exmouth).

Generally, with or without the help of computer software, you will need to sort answers into analytical categories in order to undertake content analysis of the different points made. In practice this meant that every statement has to be analysed for content and placed under an appropriate heading, along with any others which are sufficiently similar. These may then be grouped under more general umbrella headings to produce the description of points made with reference to their nature, range and frequency. For an example drawn from the Innovations in Teaching and Learning in Higher Education project, see C. Examples, below.

A description of 'Interpretative Phenomenological Analysis' as a method for analysing interview data may be found at: http://www.jisc.ac.uk/media/documents/programmes/elearning_pedagogy/lex_method_final.doc. To see how findings from such an analysis are presented visit: http://www.jisc.ac.uk/uploaded_documents/LEX%20Final%20Report_August06.pdf.

11. How do I write up the findings?

I recommend combining the content analysis of responses in terms of their nature, range and frequency (quantitative) with illustrations drawn from the data of particularly significant examples (qualitative). If you just give the latter you may be accused of 'cherry picking', ie of selecting quotations designed to reinforce your case. Of course, interview transcripts or notes are by their nature bulky and you may not be able to give them all in your research report. However, you may be able to give the data collected in one more particularly significant interview in your appendices so that the reader has a chance of checking your interpretations and selections against their source. For more help, consult the RESINED component on Writing Up Research.

12. What should I avoid?

Ted Wragg (1978, p 20) gives the following list of stereotypes to avoid:

The ESN Squirrel Collects tapes of interviews as if they are nuts, only does not know what to do with them other than play them back on his Hi-Fi.

The Ego-Tripper Knows in his heart that his hunch is right, but needs a few pieces of interview fodder to justify it. Carefully selected quotes will do just that, and one has no idea how much lies on the cutting room floor.

The Optimist Plans 200 interviews with a randomly selected group of secondary school Heads by Xmas. Is shortly to discover 200 synonyms for 'get lost'.

The Amateur Therapist Although ostensibly enquiring into parents' attitudes to lacrosse, gets so carried away during interview he tries to resolve every social/emotional problem he encounters. Should stick to lacrosse.

The Guillotine Is so intent on getting through his schedule he pays no attention to the answers ands chops his respondents short in mid-sentence. (He actually does manage to do 200 interviews by Xmas.)

C. Examples of interview schedules (lists of questions or topics to discuss)

Click on the examples given below:

Market research interview (extract)

Structured interview (extract)

Schedule for semi-structured interview from the Innovations in Teaching and Learning in Higher Education project

Agenda for a focus group discussion from the Innovations in Teaching and Learning in Higher Education project

Example of analysis from the Innovations in Teaching and Learning in Higher Education project

Also, The Question Bank at http://qb.soc.surrey.ac.uk/ has many examples of lists of questions used in structured interviews, which it calls 'questionnaires'. It's a good place to look for examples, which you can normally use without worrying about copyright (although you will need to acknowledge the sources in the normal fashion).

D. Tasks

Tasks, once completed, should be sent to resined@plymouth.ac.uk, making clear:

which component it is from;

which task it is (A, B or C);

the name of your dissertation supervisor.

It will then be passed on to the component leader (and copied to your supervisor). The component leader will get back to you with comments and advice which we hope will be educative and which will help you in preparing your dissertation proposal once you are ready. (Remember that these tasks are formative and that it is the proposal which forms the summative assessment for the MERS501 (resined) module.) This email address is checked daily so please use it for all correspondence about RESINED other than that directed to particular individuals for specific reasons.

TASK A (THE NATURE OF EDUCATION RESEARCH)

Make a case for choosing interviews as a data collection technique for your own research project, making reference to the literature on research methodology (see section E. Further Reading for some suggestions). Consider possible criticisms of your choice. Make sure you reference your sources correctly.

TASK B (DATA COLLECTION)

Design your own interview 'schedule' (a list of questions or topics to discuss) on a topic of your choice, describing the target group and providing a draft ethics protocol (getting informed consent, being honest, not forcing anyone to take part, giving participants the right to withdraw, protecting them from harm, feeding back the results, maintaining confidentiality). I STRONGLY RECOMMEND that the 'ethics protocol' for which you seek approval should take the form of the document you intend to issue to potential participants in order to obtain informed consent, which would cover issues such as confidentiality, right to withdraw, feedback, etc as appropriate. The examples referred to in the Ethics section in the 'Beginning Research' component are statements of this kind.

TASK C (DATA ANALYSIS)

When you have received feedback from the component tutor, pilot your interview with a group of at least three respondents and record the responses in whatever way you think best.

Undertake an analysis of the data (with or without the help of a computer).

E. Further Reading

CD-ROM

Barrett, Elizabeth; Lally, Vic; Purcell, S & Thresh, Robert (1999) Signposts for Educational Research CD-ROM: A Multimedia Resource for the Beginning Researcher. Sage Publications, London.

Find the section on Interviews, by going to the 'Departure Lounge', clicking on 'Ready to Depart', then 'Travelogues' (on the world map) and finally 'Interviews'.

Bell, J (1999) Doing Your Research Project (3rd edition), Buckingham, OUP - for details click here

Clough, P & Nutbrown, C (2002) A Student's Guide to Methodology, London, SAGE

Cohen, L ; Manion, L & Morrison, K (2000) Research Methods in Education (5th edition), London, RoutledgeFalmer - for details click here

Denscombe, M (2003) The Good Research Guide: 2nd edition, Buckingham, OUP - for details click here

Drever, E (1995) Using Semi-Structured Interviews in Small-Scale Research: A Teacher's Guide, Edinburgh, Scottish Council for Research in Education. For details visit: http://www.scre.ac.uk/cat/res_guides.html

Mason, J (2002) Qualitative Researching, London, Sage (chapter 4 contains some useful advice for interview design and how to ensure that the questions you use are consistent with the research design and theoretical framework you employ).

Pollard, A (1985) The Social World of the Primary School, London, Cassell.

Radnor, H (1994) Collecting & Analysing Interview Data, University of Exeter, Research Support Unit, School of Education.

Scheurich, J J (1995) A postmodernist critique of research interviewing, Qualitative Studies in Education, 8, 3, 239-252.

Wengraf, T (2001) Qualitative Research Interviewing, London, Sage. For details search by author at: http://www.sagepub.co.uk/

Wilson, V (1997) Focus Groups: a useful qualitative method for educational research? British Educational Research Journal, 23, 2, 209-224.

Wragg, E C (1978) Conducting and Analysing Interviews, Nottingham University School of Education, TRC-Rediguides.

Observation Techniques

Originally prepared by Professor Andrew Hannan.

Component now led by Dr. Peter Kelly.

© A Hannan, Faculty of Education, University of Plymouth, 2006

CONTENTS

A. Introduction

B. Examples

C. How to do it

participant or non-participant

gaining and maintaining access

ethics and confidentiality

ethnographic methods (field notes, generating insights, the nature of evidence, progressive focusing, presentation of findings)

practical advice on ethnographic observation

D. Tasks

E. Further Reading

This component is intended as a supplementary set of notes to be considered alongside the 'Qualitative Research' component by Professor Peter Woods. The 'Qualitative Research' component covers more ground and includes 'tasks' for MA Dissertation students (UoP). To visit click here.

A. Introduction

1) Observational techniques are an important aspect of many action research studies and of case studies whether undertaken by participants or outsiders.

2) In a way all of us are already well practised in the arts of observation - we all need to observe human behaviour in our personal and professional lives, we are all familiar with the need to come to conclusions based on our observation, to generate explanations and understandings and even to come up with predictions.

3) It is important that we attempt to build on those skills that we already possess and to exploit those aspects of common sense that are of benefit.

4) However, in research we need to go beyond the subjective and impressionistic, we need to be aware of and, if possible, eliminate bias, we need to be systematic and open about our procedures so as to open them up for public scrutiny so that others may check the bases on which we reach conclusions. I shall discuss later the various approaches that might be adopted in pursuit of these aims.

5) Let me go back to a more basic question, what specific contribution can observation make to research in education? What can observational techniques provide that experiments, surveys, questionnaires or interviews can't?

Write down your own response before referring to mine, by clicking here.

6) There are those who argue that even observation should attempt to treat social interaction purely in terms of externally observable behaviour with no attempt being made to impute motive other than a consideration of the stimuli which may have occasioned certain responses. One variant of this approach is that popularised by Desmond Morris in his books such as Manwatching (1978) that attribute all human behaviour to underlying drives and patterns either biologically or socially inherited as a product of evolutionary pressures.

7) Some clinical or pseudo-clinical studies in the behaviour modification tradition are only interested in observing external behaviour where the intrinsic meaning of action to the actor is not seen as an important aspect of the analysis, eg with references to 'speech acts' or 'writing behaviour'.

8) In contrast to this are approaches that start from the assumption that social action is the product of intended and meaningful behaviour that makes reference to understandings of a shared social world.

9) For those who take such an interpretative approach, typical of symbolic interactionism, phenomenology and ethnomethodology, observation is the key method of research, often allied to open or semi-structured interviews.

B. Examples

1) Let us look at the sorts of observational techniques available and at the contexts within which they are most applicable.

2) If we wanted to find out if tutors habitually favoured male or female students with more questions, more positive responses, more attention, or whatever, we could gather the information in a fairly straight forward way by counting aspects of tutor student interaction and differentiating between males and females. We could then, perhaps, attempt to intervene by training tutors to interact in a non-gendered way and then go back to observing their teaching behaviour and in a similar way attempt to count the interactions to see if the change had been successful. Such observation could be based on a simple check list, with the observer ticking against the name of each student whenever the tutor directly interacted with him or her and then breaking down the nature of that interaction with a further tick into one of a number of boxes such as 'praise', 'reprimand', etc. This would allow the researcher to compute the percentages of negative and positive interactions for males and females.

What would be the limitations of this method? What would it be unable to tell us? Is it purely objective or does it require some subjective judgement and interpretation of intent and meaning?

Write down your own response before referring to mine, by clicking here.

3) One of the points at issue here is whether it is best for observers to immerse themselves in 'the field' first and then generate categories to order their at first unstructured observations, or whether it is best to walk in with research 'instruments' designed by other researchers which can then be applied in a seemingly unproblematic manner. Which is the best technique depends on the nature of the research undertaken, the former method being better for exploratory case studies and the latter for large-scale surveys, or as a means of providing wider reference points for more qualitative small-scale enquiry.

4) The most famous example of a structured observation schedule for classroom research is the Flanders Interaction Analysis Category System (FIAC, see Flanders, N, 1970, Analyzing Teaching Behaviour) which was used in an adapted fashion by the ORACLE researchers looking at primary classrooms in the UK (see Galton, M and Simon B, eds., 1980, Progress and Performance in the Primary Classroom, Galton, M et al, 1999 Inside the Primary Classroom: 20 Years On, Simon, B and Willcocks, J, 1981, Research and Practice in the Primary Classroom). For a good description of the methods involved and examples of schedules used see Croll, P, 1986, Systematic Classroom Observation.

5) Other researchers prefer to rely on recording what goes on, in the classroom and other arenas, using videos, still photography, tape recorders, verbatim or selective field notes or memory, or a combination of one or more of these. The difficulty with all these methods is that the problem of what to do with all the data so collected is far more pressing than in the case of the structured observation schedules where the categories are pre-ordained and the task is simply one of computation and statistical analysis.

6) The problem is compounded by the more sophisticated the technology in that the more that is recorded the more immense the task of analysis, which often involves the very time consuming business of transcription. The data are certainly richer and more faithful to the nature of the interaction, but the researcher often resorts to pretty crude devices of content analysis faced by the seeming infinity of what has been observed. Reality is multi-layered and 'what happened' is the product of a wide range of influences and contributors and is open to a variety of interpretations from different perspectives.

7) Again, the researcher has to be guided by the nature of the research focus in order to home in on the important information, although it is not an easy task to separate the incidental from the essential.

8) Thus, in my own PhD research, I was interested in the ways in which issues came to be defined as 'problems' for the school I was studying, how those problems led to 'conflicts' between staff and the impact those conflicts had on school 'policy'. I could have merely interviewed or questionnaired staff on such issues, but I decided that such devices were necessary but not sufficient. I needed to observe the process in action, how the problems arose, how the discussion of them reflected different staff interests and ideologies and the ways in which decisions were made within a hierarchically structured process which brought about changes in school policy. In short, I was interested in what Stephen Ball has come to call The Micro-Politics of the School (l987). I therefore spent time observing in classrooms, hallways, and staff rooms, and, crucially, in staff meetings where decisions were made which became school policy and came to structure events in the school as a whole.

9) The observation of such contexts provided a means of seeing the school dynamically and of finding out about people's views as expressed actively within the micro-political world of the decision-making process, rather than passively in response to an interview or questionnaire question. I would argue that such devices, or a combination of such techniques, gave me access to much more of the reality I was attempting to research.

C. How to do it

Let me focus on a number of aspects, viz:

participant or non-participant

gaining and maintaining access

ethics and confidentiality

ethnographic methods (field notes, generating insights, the nature of evidence, progressive focusing, presentation of findings)

practical advice on ethnographic observation

1) Participant or non-participant.

As a participant you are an actor with a certain amount of power and this will inevitably affect how others see you and how you see them. It is therefore important that you try and gauge the extent to which your observations may be structured by your participation, eg is it impossible for you as a teacher/tutor to get real access to the pupils'/students' perspectives? You need to seek ways of overcoming such restrictions, eg getting pupils/students to observe one another (see an example using primary age pupils in Pollard, A, 1985, The Social World of the Primary School, chapter 4). You need to get others to check your observations and interpretations, e.g. by triangulation (seeking evidence from other sources uninfluenced by you such as documents, or gathering data through other devices ensuring anonymity such as questionnaires, or by taking your field notes and analyses back to the participants and asking them to comment on the accuracy of your findings).

The more you can make the taken-for-granted world of your own actions strange, the better. This is best done by widening your field of observation to look at other cases or by getting others to look at what you do. What's commonplace to you is strange for others and vice versa. It may seem preferable to do your research purely as an outsider but this is often more difficult than it seems because the presence of any observer is part of the scene and inevitably influences behaviour (which is even true of impersonal recording devices). What we have here, though, is a matter of degree. The choice is between 'going native' and maintaining distance. The trade off is between the advantages of shared perspective likely to be gained from full participation in role and the disadvantages of failing to make problematic the taken-for-granted assumptions made by actors in that world.

There are numerous studies of education which illustrate the different resolutions of this dilemma available, e.g. Willis, P, 1977, Learning to Labour, Patrick, J, 1975, A Glasgow Gang Observed, Hargreaves, D, 1967, Social Relations in a Secondary School, Lacey, C, 1970, Hightown Grammar. There is a very good discussion of such issues in Woods, P, 1986, Inside Schools.

There are also some classic American studies of the medical world making use of participant and other forms of observation, such as Goffman, E, 1968, Asylums: Essays on the Social Situation of Medical Patients and Other Inmates, Strauss, A et al, 1964, Psychiatric Ideologies and Institutions, and Becker, H et al, 1961, Boys in White: Student Culture in Medical School. For a British example, see Atkinson, P 'In cold blood: bedside teaching in a medical school' in Chanan, G and Delamont, S, eds, 1975, Frontiers of Classroom Research.

The Becker and Atkinson references above concern a particular field of Higher Education (medical schools). Becker, H et al, 1968, Making the Grade: The Academic Side of College Life is a case study of a US college that focuses on student perspectives.

2) Access - getting in and staying in.

Generally speaking, people are quite flattered by attention and this sometimes overcomes any inhibitions so that matters quite secret are paraded before the observer with seeming abandon. At other times, particularly where an issue has been the subject of recent press attention, the shutters go up and there is no way in. All you can do in such circumstances is to give up and try something else likely to be more productive - you haven't the time to spend on lengthy negotiation. Crucial to the business of initially gaining access is the whole matter of assurances of confidentiality, anonymity, etc, which I consider below. Your stance, once granted access, depends on how much of an active participant you want to be in the arena you are observing - the more obviously committed you are to one particular stance or ideology the more others will take this into account when revealing their thoughts or their actions to you. If all involved can accept you without risk to their interests you get maximum access, but this may involve you in compromising your principles, if not breaking the law!

3) Ethics and Confidentiality.

Like all other issues to do with morality, the debates here are endless. You need to comply with the University of Plymouth requirements (see the section on ethics in the Beginning Research component). I recommend promising as much anonymity as possible and pledging as much confidentiality as necessary for the research to go ahead without compromising its objectives. Disguising the institution, staff and students from outsiders is a lot easier, however, than doing so internally. Be careful about giving the participants a veto over what you write; try and get them to accept anonymity or a right of reply or comment instead. The biggest dilemma for the observer, per se, is what to do about what he/she observes, particularly when he/she has decided to adopt a neutral stance and an attempt is being made to obtain or protect access. Decisions have to be made about the relative importance of the study vis-à-vis other moral considerations. This is particularly true for a teacher as observer who is privy to acts of pupil misbehaviour!

4) Ethnography.

The principal method employed by all ethnographers is the collection of field notes. These may include both direct observation and reflections on what has been observed. Again, Woods (l986) is very helpful on what is involved. I'd recommend a daily record of things observed, including records taken at the same time as the events where possible, and comments written about their significance as well as decisions made by the observer on methodology and any theoretical insights generated, tested, confirmed, disproved. The process of generating insights is a difficult one, in that it requires the researcher to be analytical. In ethnographic observation the researcher is the research instrument and he/she must make decisions about how and what to sample and about issues which currently require attention. The inductive method is all about coming up with ideas from the data which are then explored further in different contexts (see Glaser, B and Strauss, A, 1967, The Discovery of Grounded Theory). The idea is that theories, hypotheses, insights, should emerge from the observations, so that they are grounded in observed experience. This may involve a process of progressive focusing where the observer begins to sort out the peripheral from the central factors involved and directs his/her attention to looking at key contexts for the vital evidence. Ethnographic accounts do not rely on statistical tables, although it is possible to make use of computers to help in the organisation of data and the process of generating insights, looking for patterns, etc. (For details of current qualitative data analysis software visit http://www.soc.surrey.ac.uk/caqdas/. N6 and NVivo (QSR) are available to all those who are registered University of Plymouth users via the file server. Both of these come with a built-in tutorial that will teach you how to use them.)

Writing up ethnography is very like telling a story, except that the writer must make apparent the devices used to construct the account and must provide evidence for the conclusions reached. That evidence needs to be presented in a way which goes beyond a mere illustration of the points made. The reader has to be given access to material that allows him/her to check the interpretations used by the observer. This may involve the presentation of great chunks of field notes, and often necessitates the sort of triangulation mentioned earlier. Often, the main body of the research report includes snippets from the field notes whereas more representative longer chunks detailing crucial instances are given in an appendix. All the time the ethnographer has to self-consciously justify the account offered. A crucial chapter in any ethnographic account is a description of the methodology employed - those wishing to adopt such an approach should keep field notes about their own methodological dilemmas and choices. At least ethnography is aware of its limitations, it has to make a claim to relevance, it has to substantiate its right to be listened to. Other methods of research often fail to make as explicit the bases of their claims to knowledge.

5) Practical advice on ethnographic observation

There follows an extract from Research Communiqué number 29 from Denis Hayes (University of Plymouth, September 2006), which draws on Delamont (2002).

With respect to observational data Delamont describes it as her favourite kind of data collection and regrets the fact that owing to time pressures she has to spend more time interviewing and examining documents than simply sitting and watching. Her enthusiasm for this aspect of research is reflected in the fact that she allocates nine pages of text to the subject (pp. 130-8) subdivided into:

What to look at

How to observe

Where and when to look

What to record

In terms of what to look at Delamont accepts that unless the research project has pre-specified objectives it is difficult to be precise, though she suggests that following an initial period of relatively unfocused watching it is essential to start paying close attention to a selective set of phenomena (p. 130/1). In an ethnographic study it may be appropriate to observe clothing worn by participants, furniture arrangement, items employed and used, etc. Over the period of observation some indication about time spent, physical movement of players, handling of equipment/resources and recording verbatim speech (as far as possible) is often desirable. In an important sense it does not matter what the observer looks at, as long as the gaze is focused on some person, object or location in a thoughtful, principles way (p. 132).

In terms of how to observe, the author refers to work by Wolcott (1981), commending it as the best paper she knows on how to observe. Wolcott proposes four strategies for deciding what to look at and how to look:

Observations by broad sweep

Observations of nothing in particular

Searching for paradoxes

Searching for the problem(s) facing the group

The broad sweep approach usually has two outcomes: first, it makes the researcher aware of the need for selectivity; second, it makes the researcher aware of what really matters to him/her
The observations of nothing in particular approach is based on ‘wait and see what jumps out’ and is likened to watching for a blip on a radar screen that indicates unusual activity [You will know it when you see it!]

The two ‘searching’ approaches are useful for fighting familiarity. Delamont offers an example of a paradox in a study that found that children drew more during ordinary lessons than they did during art lessons! An example of working out participants’ problems is found in studies of medical students. Wolcott also mentions a fifth approach in which the researcher is trained in the techniques used by participants as away to better understand events and knowing how to look.

Where and when to look
Delamont comments that choosing where and when to look is also a matter of systematic, principles, reflexive decision-making (p. 134). She draws from a study on how children with learning difficulties were integrated into ordinary schools. Observations took place in three contexts:

Where pupils were free to choose both their activities and their companion (e.g. playground)

Where large groups of children were relatively free to mingle within broad categories (e.g. assembly; lunch sittings; communal singing)

Where small groups of pupils were in close proximity (notably in lessons)

Other key places for observation may include corridors and staff rooms. She makes special note of the prevalence of ‘rhetorical divisions’ between members of the same group; folklore; jokes; narrative; atrocity stories; serious educational discussion; barbed comments. The alert researcher does not neglect the urban legends or other folklore in playground and staff room (p. 137).

Finally, in terms of what to record Delamont recommends that recording during observation should be carried out as unobtrusively as possible, ideally noting verbatim speech or at least some key words/phrases that will serve to jog the memory later. Nevertheless, ten minutes of good observation, well written up is worth an hour’s notes lying forgotten in an unopened notebook (p. 138). She reminds the reader that it is essential to constantly analyse and interpret data; or it may become so complex, convoluted and confusing that it is fails to serve any purpose. You have been warned!

D. Tasks

Tasks, once completed, should be sent to resined@plymouth.ac.uk, making clear:

which component it is from;

which task it is (A, B or C);

the name of your dissertation supervisor.

TASK A (THE NATURE OF EDUCATION RESEARCH)

Make a case for choosing observation as a data collection technique for your own research project, making reference to the literature on research methodology (see Section E for Further Reading). Consider possible criticisms of your choice. Make sure you reference your sources correctly.

TASK B (DATA COLLECTION)

Design your own observation technique for your research topic, describing and discussing the aims of the observation, the nature of the observation (what/who/how/when will you observe?) and the method of recording (eg field notes or using a structured observation schedule). You will also need to provide a draft ethics protocol (getting informed consent, being honest, not forcing anyone to take part, giving participants the right to withdraw, protecting them from harm, feeding back the results, maintaining confidentiality). I STRONGLY RECOMMEND that the 'ethics protocol' for which you seek approval should take the form of the document you intend to issue to potential participants in order to obtain informed consent, which would cover issues such as confidentiality, right to withdraw, feedback, etc as appropriate. The examples referred to in the Ethics section in the 'Beginning Research' component are statements of this kind.

TASK C (DATA ANALYSIS)

When you have received feedback from me, pilot your observation and record data.

Consider, in relation to the literature, how you will analyse your data.

Undertake an analysis of the data and send it to us at the resined email address, as above.

E. Further Reading

NB See especially those items highlighted in bold

CD-ROM

Barrett, Elizabeth; Lally, Vic; Purcell, S & Thresh, Robert (1999) Signposts for Educational Research CD-ROM: A Multimedia Resource for the Beginning Researcher. Sage Publications, London.

Find the section on Observation, by going to the 'Departure Lounge', clicking on 'Ready to Depart', then 'Travelogues' (on the world map) and finally 'Observation'.

Atkinson, P. (1975) In cold blood: bedside teaching in a medical school, in Chanan, G and Delamont, S, eds, Frontiers of Classroom Research.

Ball, S. (1987) The Micro-Politics of the School.

Becker, H; Geer, B.; Hughes, C. and Strauss, A. (1961) Boys in White: Student Culture in Medical School.

Becker, H; Geer, B. and Hughes, C. (1968) Making the Grade: The Academic Side of College Life.

Croll, P. 1(986) Systematic Classroom Observation, (see pages 84-93 on 'Observing in Classrooms').

Delamont, S. (2002) Fieldwork in Educational Settings.

Flanders, N. (1970) Analyzing Teaching Behavior.

Galton, M. and Simon B., eds., (1980) Progress and Performance in the Primary Classroom

Galton, M; Hargreaves, L.; Comber, C.; Pell, T.; and Wall, D, (1999) Inside the Primary Classroom: 20 Years On.

Glaser, B. and Strauss, A. (1967) The Discovery of Grounded Theory.

Goffman, E. (1968) Asylums: Essays on the Social Situation of Medical Patients and Other Inmates.

Hargreaves, D. (1967) Social Relations in a Secondary School.

Hopkins, D. (1993) A Teacher's Guide to Classroom Research (see Chapter 7 on 'Methods of Observation in Classroom Research').

Lacey, C. (1970) Hightown Grammar.

Morris, D. (1978) Manwatching.

Patrick, J. (1975) A Glasgow Gang Observed.

Pollard, A. (1985) The Social World of the Primary School (see especially chapter 4).

Simon, B. and Willcocks, J. (1981) Research and Practice in the Primary Classroom.

Simpson, M. and Tuson, J. (1995) Using Observations in Small-Scale Research: a beginner's guide, SCRE. For details visit: http://www.scre.ac.uk/cat/res_guides.html

Strauss, A.; Schatzmann, L.; Bucher, R.; Ehrlich, D. and Sabshin, M. (1964) Psychiatric Ideologies and Institutions.

Willis, P. (1977) Learning to Labour.

Wolcott, H. F. (1981) ‘Confessions of a trained observer’, in T. S. Popkewitz and B. R. Tabachnik (Eds.) The Study of Schooling.

Woods, P. (1986) Inside Schools.

Quantitative Methods in Education Research

Prepared by Professor John Berry

© J Berry, Centre for Teaching Mathematics, University of Plymouth, 2005

(links updated August 2006)

CONTENTS

A. INTRODUCTION

B. QUANTITATIVE AND QUALITATIVE RESEARCH

C. INGREDIENTS OF QUANTITATIVE RESEARCH

D. STATISTICS

E. STATISTICAL CONCEPTS & QUANTITATIVE PROCEDURES

F. TASKS

G. FURTHER READING

H. WEBSITES

A. INTRODUCTION

This component is unable to do more than help you to begin thinking about quantitative methods in educational research. Its aim is to give you an insight into the issues should you choose quantitative methods as part of your research methodology.

We will briefly address the following questions:-

What are quantitative methods?

What are the ingredients of quantitative methods?

How do you go about research design?

Education research has moved away largely from the numbers approach in recent years, and the emphasis has been on qualitative methods. However, the use of numbers can be a very useful tool, either as part of a larger project that employs many different methods or as a basis for a complete piece of work. With the use of sophisticated software packages such as SPSS it is relatively easy to deal with the computation side of things and it is possible to come up with numerous tables and charts almost instantly once your data is installed. However, it is very important that the underlying principles of statistical analysis are understood if sense is to be made of the results spewed out by such a package in terms of your research.

This component consists of two sections; we begin with an overview of quantitative methods and finish with a brief introduction to some of the basic statistical concepts to be looking for when you read research papers that use quantitative methods of research.

Back to CONTENTS list

B. QUANTITATIVE AND QUALITATIVE RESEARCH

In simple terms we can think of two approaches to investigations in educational research: qualitative and quantitative. In the former we use words to describe the outcomes and in the latter we use numbers.

Quantitative research methods were originally developed in the natural sciences to study natural phenomena. However examples of quantitative methods now well accepted in the social sciences and education include:

surveys;

laboratory experiments;

formal methods such as econometrics:

numerical methods such as mathematical modelling.

Qualitative research methods were developed in the social sciences to enable researchers to study social and cultural phenomenon. Examples of qualitative methods include:

action research aims to contribute both to the practical concerns of people in an immediate problematic situation and to the goals of social science by joint collaboration within a mutually acceptable ethical framework;

case study research - a case study is an empirical enquiry that investigates a contemporary phenomenon within its real-life context;

ethnography- the ethnographer immerses her/himself in the life of people s/he studies and seeks to place the phenomena studied in its social and cultural context.

Other components of this module cover various qualitative research methods.

Structure of Research Papers

When setting out on educational research you will be (have been) encouraged by your supervisor to read appropriate publications and this is a good way of identifying the methods of research that seem most used in your research area. A typical structure for a research paper is summarised in the table below:

literature survey	other people’s work
methodology	qualitative or quantitative
results	your work
discussion/conclusions	your discussion and reference to others

Table 1 Structure of typical research paper

Activity 1

Scan read the following three papers:

Mark Cosgrove: A study in science-in-the-making as students generate an analogy for electricity. International Journal of Science Education. 17 No 3, pp 295-310, 1995

Susan Picker: Using Discrete Mathematics to give Remedial Students a Second Chance. DIMACS, 36, pp 35-41, 1997

John Berry and Pasi Sahlberg: Investigating Pupils' Ideas of Learning. Learning and Instruction. 6 No 1, pp 19-36, 1996

Identify the research method being used in each paper. Answer the question 'Why use numbers in education research?' with reference to these examples.

Back to CONTENTS list

C. INGREDIENTS OF QUANTITATIVE RESEARCH

As part of your research you will be looking at certain characteristics (variables) and endeavouring to show something interesting about how they are distributed within a certain population. The nature of your research will determine the variables in which you are interested. A variable needs to be measured for the purpose of quantitative analysis.

We may collect data concerning many variables, perhaps through a questionnaire, or choose to measure just two or several variables by observation or testing. The variables we are interested in may be dependent or independent. There will be other features present in the problem that may be constant or confounding.

Using the data that you have collected then you can:

Describe variables in terms of distribution: frequency, central tendency and measures and form of dispersion. Descriptive statistics include averages, frequencies, cumulative distributions, percentages, variance and standard deviations, associations and correlations. Variables can be displayed graphically by tables, bar or pie charts for instance.
This may be all the statistics you need and you can make deductions from your descriptions. In fact univariate (one variable) analysis can only be descriptive.
But descriptive statistics can be used to describe a significant relationship between two variables (bivariate data) or more variables (multivariate).

Infer significant generalisable relationships between variables. The tests employed are designed to find out whether or not your data is due to chance or because something interesting is going on.

See the section on Variablesin The Research Methods Knowledge Base.

Often it is not possible to undertake a true experiment as part of your research and a common research approach in educational research is called quasi-experimental design represented in the following diagram:

Experimental Group	O1	X	O2
Control Group	O3		O4

In this figure O1 and O3 represent initial testing of the two groups; X represents some intervention or experimentation strategy with one of the groups and O2 and O4 represent final testing of the two groups. We would use the test results to investigate whether the experimental teaching approach has led to an improvement in the feature being tested.

Back to CONTENTS list

D. STATISTICS

Perhaps the best way to begin to appreciate the kind of statistics that you might employ in your own research is to have a look at what others have done.

Read the paper by John Berry and Pasi Sahlberg: Investigating Pupils' Ideas of Learning. Learning and Instruction. 6 No 1, pp 19-36, 1996 and attempt to identify the statistics that are used.

The section ‘Findings’ on pages 28 – 32 contains some of the important measures that we use in quantitative research methods. The mean and standard deviation tell us about the average and spread of the data. The bar graphs on page 31 allow a visual comparison of the means between the two groups of samples.

The symbol ‘p’ represents the probability of a significant difference between the two groups. This is probably the most difficult concept to grasp because in some senses it is counter intuitive. The probability of an event happening lies between 0 and 1. A large probability (i.e. p close to 1) implies a high likelihood of the event happening. For example if you are told that there is a 95% chance of winning a game (p = 0.95) then put your money on winning! On the other hand if there is a 5% chance of you winning (p = 0.05) it’s probably best not to bet on yourself!

So if p is small (close to 0) the event is less likely to happen than if p is large (close to 1). Let’s see what this means in the context of educational research. As an example look at page 31 Berry and Sahlberg. If we compare the means of the scores of the UK and Finland pupils for the statement "I learn better by doing work by myself than by watching the teacher", then there is a probability of less than 0.001 (i.e. 0.1%) that the difference in the means will occur by chance. In ordinary language the probability of it happening by chance is so small that we say it is a significant result. Because it is unlikely to occur, the reason that it does is significant.

In our analysis we look for small probabilities usually less than 5% or p = 0.05. Then we say that the result is significant at the 5% level of significance.

There is also evidence from Figure 3 that there is some difference between the UK and Finland pupils. For Q2 the UK rating is positive and the Finland rating is negative.

Very often a comparison of the means as in Figure 3 is a clue. For example look at Q7 in Figure 3. For the UK pupils the rating is positive whereas for the Finland pupils it is negative. The ‘p – value’ for this feature is less than 0.001 (0.1%) and so we conclude that there is a significant difference between the two groups of pupils for the statement "I like most teachers in my school".

One of the first steps in the design of a piece of quantitative research is setting up what is called a hypothesis. For example, we might propose the hypothesis that there is no difference between the UK pupils and Finnish pupils views of their teachers (item 7 in Berry and Sahlberg). This is called the null hypothesis. We then need to use the pupil responses to try to disprove this hypothesis.

To be able to compare quantities we need to define a statistic whose distribution is known. In the paper by Berry and Sahlberg the t-statistic is used as a measure of the difference between the means of the two groups of pupils. Having calculated the value of the t-statistic for the feature under investigation we then look up in tables the probability of the feature occurring. (At this stage don’t worry about how it is calculated!) You can see on page 31 the t-statistic and its associated p-value.

At the heart of quantitative research methods is some very sound statistical theory. If you are planning to carry out a research investigation using quantitative research methods you do not need a thorough grounding in this theory but you will need an understanding of the statistical methods. We use statistical software packages to do the arithmetical calculations so the important skill is not doing the mathematics but is interpreting the results.

In what follows we have gathered together some of the essential statistical ideas needed for quantitative research. It is a summary with some examples to provide a flavour of the ideas.

Back to CONTENTS list

E. STATISTICAL CONCEPTS & QUANTITATIVE PROCEDURES

NB What follows is merely an introductory overview of some of the relevant concepts and procedures. To find out more go first to a general textbook such as Denscombe (1998), Chapter 10, and then, for a much fuller account, try Peers (1996).

1. Variables

numerical measurements: person’s age or weight; size of a family

non-numerical measurements: position on a scale indicating a level of agreement e.g. Likert rating scale

continuous data: measurement that can, in principle, take any value within a certain range e.g. time, age and weight

categorical data: (or discrete data): measurements that can take only known discrete values e.g. the number of rooms in a house, the number of children in a family

nominal data: numerical values are assigned to categories as codes e.g. in coding a questionnaire for computer analysis, the response ‘male’ might be coded as ‘1’; and ‘female’ as ‘2’. No mathematical analysis is usually possible and no ordering is implied.
ordinal data: numerical values are assigned in accordance with a qualitative scale e.g. in coding a questionnaire for computer analysis, the responses ‘very good’, ‘good’, ‘poor’ and ‘very poor’ are coded ‘4’, ‘3’, ‘2’ and ‘1’ respectively.

See also the section on Variables in The Research Methods Knowledge Base.

2. Basic Measures

mean: is a measure of the central location or average of a set of numbers, e.g. the mean of 2 7 2 1 8 2 6 9 10 5 1 4 is 4.75

standard deviation: is the square root of the variance!!

variance: is a measure of dispersion (or spread) of a set of data calculated in the following way:

median: is the centre or middle number of a data set, e.g. the median of 2 7 2 1 8 2 6 9 10 5 1 4 is 4.5

quartiles: divide a distribution of values into four equal parts. The three corresponding values of the variable are denoted by Q₁, Q₂ (equal to the median) and Q₃

range: is a measure of dispersion equal to the difference between the largest and smallest value.

3. Frequency Distribution

Example A

A new hybrid apple is developed with the aim of producing larger apples than a particular previous hybrid. In a sample of 1000 apples, the distribution of weights was as follows:

Weight (g)	0-50	50-100	100-150	150-200	200-250	250-300	300-350	350-400
frequency	20	42	106	227	205	241	106	53

Apples can only be sold to a particular supermarket with a weight greater than 150g. What proportion of the new hybrid apples would be rejected by this supermarket.
How many grams above this weight of 150 g is the mean weight of apples?
What is this difference in weights in units of the standard deviation of apple weights?

Suppose that we graph the data using columns to show the amount in each group. We get a frequency distribution.

From the data there are 168 apples whose weight is less than 150 g and 832 apples whose weight is greater than 150 g. There are 1000 apples altogether.

We can deduce that the proportion = 16.8 % cannot be sold to the supermarket.

The mean weight of the apples is 223.35 g and the standard deviation is 78.9 g.

The difference in weight between 150 g and the mean is 73.35 g and this is of a standard deviation.

4. Measures of Location and Dispersion

A distribution is symmetrical if the difference between the mean and the median is zero. An appropriate pictorial representation of the data, (histogram, stem and leaf diagram etc.) would produce a mirror image about the centre:

A distribution is positively skewed (or skewed to the right) if the mean - median is greater than zero. Such data when represented by a histogram would have a right tail that is longer than the left tail:

A distribution is negatively skewed (or skewed to the left) if the mean - median is less than zero. Such data when represented by a histogram would have a left tail that is longer than the right tail:

If data are skewed then the best measure of location is the median and the best measure of dispersion is the interquartile range.

If data are symmetrical then the best measure of location is the mean and the best measure of dispersion is the standard deviation or variance.

5. Probability

This is an important concept in statistics and is an important part of our story.

It is defined in the following way: if an experiment has n equally likely outcomes and q of them are the event E, then the probability of the event E, P(E), occurring is

P(E) =

Some simple examples:

the probability of getting a head from the toss of a fair coin is

the probability of getting a six from the throw of a fair die is

the probability of getting the ace of spades from cutting of a pack of cards is

The smaller the probability the more unlikely the event is to happen. This is an important concept in quantitative methods in education research as we shall see.

There is an important link between probability and the frequency distribution. Consider again the hybrid apple example above. We saw that the proportion of apples weighing less than 150 grams was 0.168. If we pick up one of the apples ‘at random’ then it could weigh less than 150 g and be rejected or it could weigh more than 150 g and be accepted by the supermarket.

The probability of picking such an apple is 0.168.

Exercise

To illustrate this idea further complete the following:

the probability of picking an apple in the weight range 200-250 g is
the probability of picking an apple in the weight range 350-400 g is
the probability of picking an apple with a weight greater than 300 g is

The important idea here is that the probability is associated the amount of data under the distribution graph.

6. Testing an hypothesis

There are two basic concepts to grasp before starting out on testing an hypothesis.

Firstly, the tests are designed to disprove hypotheses. We never set out to prove anything; our aim is to show that an idea is untenable as it leads to an unsatisfactorily small probability.

Secondly, the hypothesis that we are trying to disprove is always chosen to be the one in which there is no change. For example there is no difference between the two population means.

This is referred to as the null hypothesis and is labelled H₀. The conclusions of a hypothesis test lead either to acceptance of the null hypothesis or its rejection in favour of the alternative hypothesis H₁.

Hypothesis testing: a hypothesis test or significance test is a rule that decides on the acceptance or rejection of the null hypothesis based on the results of a random sample of the population under consideration.

step 1: Formulate the practical problem in terms of hypotheses. The null hypothesis needs to be very simple and represents the status quo, i.e. there is no difference between the processes being tested.

step 2. Calculate a statistic that is a function purely of the data. All good statistics should have two properties: (i) they should tend to behave differently when H₀ is true from when H₁ is true; and (ii) its probability distribution should be calculable under the assumption that H₀ is true.

step 3: Choose a critical region. We must be able to decide on the kind of values of the test statistic, which will most strongly point to H₁ being true rather than H₀. The value of the test statistic in this critical region will lead us to reject H₀ in favour of H₁; otherwise we are not able to reject H₀ in favour of H₁. We should never conclude by accepting H₀.

step 4: Decide the size of the critical region. i.e. a 1% probability of H₀ being rejected etc.

For more on this see the section on Hypotheses in The Research Methods Knowledge Base.

Example B

In an educational research programme two groups of students are taught a topic in different ways. An experimental group uses a spreadsheet to explore the topic and a control group uses a more traditional pen and paper activity. Each group contains 20 students. At the end of the topic the teacher tests the two groups on their understanding of the topic and obtains the following data:

Experimental	5	11	25	33	35	40	45	46	52	55
	56	56	57	59	69	74	75	89	92	97
Control	33	39	44	45	45	46	47	48	49	49
	53	54	54	55	58	60	61	63	65	69

Extra data:

	mean	standard deviation
Experimental	53.6	25.0
Control	51.8	9.1

Experimental Group Control Group

How would you interpret these findings?

Some analysis

The researcher might be tempted to conclude that the experiment has had little or no effect on the performance of the experimental group as judged by the means. However the large difference in standard deviations might suggest that the experimental group is much more variable in performance than the control group.

The researcher might also be tempted to deduce that the experiment has turned some of the pupils off the task. Look at the three low scores!

Suppose that we investigate the difference in the means: Let

H₀: there is no difference between the means of the two groups: m ₁ = m ₂

H₁: the score of the experimental group is greater than the score of the control group:

m ₁ > m ₂

We use a two-sample t-test to get

The p value is 0.39 (39%) so we deduce that there is not enough evidence to reject the null hypothesis. The researcher could not deduce that there was an improvement in student performance.

7. Statistical tests

t tests

In hypothesis testing, the t test is used to test for differences between means when small samples are involved. (n £ 30 say). For larger samples use the z test.

The t test can test

i) if a sample has been drawn from a Normal population with known mean and variance. (Single sample)

ii) if two unknown population means are identical given two independent random samples. (Two unpaired samples)

iii) if two paired random samples come from the same Normal population. (Two paired samples (paired differences))

Any hypothesis test can be one tailed or two tailed depending on the alternative hypothesis, H₁.

Consider the null hypothesis, H₀: m =3

A one tailed test is one where H₁ would be of the form m > 3.

A two-tailed test is one where H₁ would be of the form m ¹ 3.

Click here for more information on t-tests.

Single sample test

Let X₁, X₂, ¼ , X_n be a random sample with mean and variance s². To test if this sample comes from a Normal population with known mean m and unknown variance s², the test statistic

is used to test the null hypothesis H₀: the population mean equals m. If the test statistic lies in the critical region whose critical values are found from the distribution of T_{n, a}, H₀ is rejected in favour of the alternative hypothesis H₁. n are the degrees of freedom and for a single sample test n = n-1, and a is the significance level of the test.

Two unpaired samples

Let X₁, X₂, ¼ , X_mbe a random sample with mean and variance s_x² drawn from a Normal population with unknown mean m_x and unknown variance s_x².

Let Y₁, Y₂, ¼ , Y_n be a random sample with mean and variance s_y² drawn from a Normal population with unknown mean m_y and unknown variance s_y².

To test the null hypothesis that the two unknown population means are the same we use the test statistic

where , the estimate of the common population standard deviation.

The test statistic T is distributed T_n, where n =(m-1)+(n-1) for two unpaired samples. If the test statistic lies in the critical region whose critical values are found from the distribution of T_{n, a}, H₀ is rejected in favour of the alternative hypothesis H₁.

Example C

It is claimed that the concentration period of students doing a particular task is normally distributed with mean 44mins. A sample of 21 students were taken, and their concentration period measured. The mean time of the sample was found to be 42mins and the sample variance was calculated to be 36min. Is there any evidence at the 5% level of significance against the claim that the population mean is 44min?

Solution

Here m = 44, = 42, s = Ö 36 = 6 and n = 21.

This is a two-tailed test since we are looking for any difference.

H₀: m = 44

H₁: m ¹ 44

Since p = 0.1423 = 14.23% there is insufficient evidence to reject the null hypothesis. We therefore conclude that the population mean concentration period is 44 minutes.

Example D

A researcher investigating the effects of pollution on two rivers takes an independent random sample of fish of a certain species from each river, measures their mass in ounces and obtains the following results.

River 1	20	10	17	7	10	18
River 2	14	6	10	8	9	7	7	6	8

Test at the 5% level of significance if there is any evidence of a difference in the mean weight between the two rivers.

Solution

Assume that each sample is taken from a normal population.

Here m = 6, = 13.667, s_x² = 23.556

n = 9, = 8.333, s_y² = 5.556.

Let m ₁ be the population mean of river 1, and m ₂ be the population mean of river 2.

H₀: m ₁ = m ₂

H₁: m ₁ ¹ m ₂

Since p = 0.043 = 4.3% <>

Back to CONTENTS list

F. TASKS

(NB: only for those University of Plymouth students undertaking the ‘Research in Education’ module as part of the preparation for the submission of a MA dissertation proposal)

Tasks, once completed, should be sent to resined@plymouth.ac.uk, making clear:

which component it is from;

which task it is (A, B or C);

the name of your dissertation supervisor.

TASK A (NATURE OF EDUCATION RESEARCH)

Does research have to be quantitative to be scientific? See Cohen et al (2000), Chapter 1, and the section on Positivism and Post-Positivism in Trochim (2000), examine the terms used in this question and discuss with reference to examples.

TASK B (DATA COLLECTION)

A research group suspects that girls and boys adopt different styles of working when using ICT in mathematics lessons. Outline a quantitative research method to investigate the effect of gender on the use of calculators on children’s understanding of mathematics. What are your variables? Formulate an experimental hypothesis for the research.What features might cause problems in your research?

TASK C (DATA ANALYSIS)

Twelve students fail an examination. After a period of revision and tutorial support they resit by taking a new examination. The marks for the original examination and the resit are shown in the table below.

Student number	1	2	3	4	5	6	7	8	9	10	11	12
Examination score	30	31	20	17	25	32	35	29	30	27	32	30
Resit score	42	38	30	21	40	45	31	32	38	50	34	40

What would you do with the data now?

Back to CONTENTS list

G. FURTHER READING

Blaxter, I., Hughes C. and Tight M. (1996) How to Research. Buckingham, Open University Press.

Bryman, A. and Cramer D. (1999) Quantitative Data Analysis with SPSS 8 Release for Windows: a guide for social scientists. London, Routledge.

Cohen, L ; Manion, L & Morrison, K (2000) Research Methods in Education (5th edition). London, RoutledgeFalmer.

Denscombe, M. (1998) The Good Research Guide. Buckingham, Open University Press.

Greenfield, Tony (ed) (1996) Research Methods – Guidance for Postgraduates. London, Arnold.

Kanji, Gopal (1993) 100 Statistical Tests. London, SAGE Publications.

Peers, Ian (1996) Statistical Analysis for Education & Psychology Researchers. London, Falmer.

Plewis, Ian (1997) Statistics in Education. London, Arnold.

Robson, C. (1990) Experiment, Design and Statistics in Psychology. Middlesex, Penguin Books.

Rose, D. and Sullivan, O. (1993) Introducing Data Analysis for Social Scientists. Buckingham, Open University Press.

Another Step Ahead

Pages

Wednesday, May 5, 2010

THANK YOU

References

Chapter 3 in progress -May 2010

A Mixed Method Design

Questionnaires in Education Research

Observation Techniques

A. Introduction

B. Examples

C. How to do it

D. Tasks

E. Further Reading

Example A

Exercise

Example B

Some analysis

Example C

Example D

MORE ADVANCED READING

Followers

Blog Archive