WU Evaluating The Quality Of Qualitative Research Questions

Qualitative researchers do not speak about validity in the same terms as quantitative researchers. In keeping with the world views and paradigms from which qualitative research arises, validity, or whether the research reflects best standards of qualitative science, is described in terms of rigour, credibility, trustworthiness, and believability. Numerous articles and books focus on validity issues for qualitative research.11–,16 Similarly, there are several qualitative research designs, and each has slightly different conventions for their appropriate conduct. This Users’ guide provides an overview of the critical appraisal of qualitative research but, as with various quantitative research designs, there are variations in how rigour and validity are addressed in specific designs.

IS THE RESEARCH QUESTION CLEAR AND ADEQUATELY SUBSTANTIATED?

Before proceeding with a full fledged review of the study, readers should look for the precise question the study sought to answer and consider its relevance to their own clinical questions. The study report should clearly document what is already known about the phenomenon of interest.

IS THE DESIGN APPROPRIATE FOR THE RESEARCH QUESTION?

More than 40 unique approaches to qualitative research methods have been identified.17 Common approaches in published healthcare research include ethnography, grounded theory, and phenomenology. Other approaches include case studies, narrative research, and historical research. Traditional ethnography seeks to learn about culture from the people who actually live in that culture.18 A grounded theory approach is used to discover the social-psychological processes inherent in a phenomenon,19 whereas a phenomenological approach is used to gain a deeper understanding of the nature or meaning of the everyday “lived” experiences of people.20 References to articles that describe these different qualitative research approaches in greater detail are listed in the online version of this Users’ guide.

Qualitative approaches arise from specific disciplines and are influenced by theoretical perspectives within those disciplines. A critical analysis of a qualitative study considers the “fit” of the research question with the qualitative method used in the study.21 Although the specific criteria for proper application of each methodological approach vary somewhat, there are sufficient similarities among the approaches to discuss them in general.

WAS THE METHOD OF SAMPLING APPROPRIATE FOR THE RESEARCH QUESTION AND DESIGN?

The emergent nature of qualitative research that results from the interaction between data collection and data analysis requires that investigators not prespecify a sample for data collection in strict terms, lest important data sources be overlooked. In quantitative studies, the ideal sampling standard is random sampling. Most qualitative studies use purposeful (or purposive) sampling, a conscious selection of a small number of data sources that meet particular criteria. The logic and power of purposeful sampling lie in selecting information rich cases (participants or settings) for indepth study to illuminate the questions of interest.14 This type of sampling usually aims to cover a range of potentially relevant social phenomena and perspectives from an appropriate array of data sources. Selection criteria often evolve over the course of analysis, and investigators return repeatedly to the data to explore new cases or new perspectives.

Readers of qualitative studies should look for sound reasoning in the description and justification of the strategies for selecting data sources. Patton offers a succinct, clear, and comprehensive discussion of the various sampling strategies used in qualitative research.14 Convenience sampling is one of the most commonly used, yet one of the least appropriate, sampling strategies. In convenience sampling, participants are primarily selected on the basis of ease of access to the researcher and, secondarily, for their knowledge of the subject matter. Purposive non-probability sampling strategies include (1) judgmental sampling, where theory or knowledge points the researcher to select specific cases: (a) maximum variation sampling, to document range or diversity; (b) extreme or deviant case sampling, where it is necessary to select cases that are unusual or special in some way; (c) typical or representative case sampling, to describe and illustrate what is typical and common in terms of the phenomenon of interest; (d) critical cases, to make a point dramatically; and, (e) criterion sampling, where all cases that meet some predetermined criteria are studied (this sampling strategy is commonly used in quality improvement); (2) opportunistic sampling, where availability of participants guides on-the-spot sampling decisions; (3) snowball, network, or chain sampling, where people nominate others for participation; and (4) theory based operational construct sampling, where incidents, time periods, people, or other data sources are sampled on the basis of their potential manifestation or representation of important theoretical constructs. Participant observation studies typically use opportunistic sampling strategies, whereas grounded theory studies use theory based operational construct sampling.

Sample size is a critical question for all research studies. A study that uses a sample that is too small may have unique and particular findings such that its qualitative transferability or quantitative generalisability becomes questionable. In qualitative research, however, even studies with small samples may help to identify theoretically provocative ideas that merit further exploration. Studies with samples that are too large are equally problematic. Whereas quantitative research has specific guidelines that frame researchers’ decisions about adequate sample size, there are only general principles, reflective of judgment and negotiation, for qualitative researchers. Examination of several areas will help readers to identify the adequacy of sample size in qualitative studies. Firstly, references about the specific method used may offer some guidance. For example, sample sizes in phenomenological studies are typically smaller than those in grounded theory and ethnographic studies. Secondly, the trade off between breadth and depth in the research affects sample size. Studies with smaller samples can more fully explore a broader range of participants’ experiences, whereas studies with larger samples typically focus on a more narrow range of experiences. Thirdly, readers can review published studies that used similar methods and focused on similar phenomena for guidance about sample size adequacy. Qualitative researchers judge the adequacy of a sample for a given study by how comprehensively and completely the research questions were answered. Readers of qualitative studies are encouraged to review the researcher’s documentation of sample size and selection throughout the course of the study.

WERE DATA COLLECTED AND MANAGED SYSTEMATICALLY?

Qualitative researchers commonly use one or more of 3 basic strategies for collecting data. One strategy is to witness events and record them as they occur (field observation). Another strategy is to question participants directly about their experience (interviews). Finally, researchers may review written material (document analysis). Readers should consider which data collection strategies researchers used and whether these strategies would be expected to offer the most complete and accurate understanding of the phenomenon.

Regardless of the strategy, the approach to data collection must be comprehensive to avoid focusing on particular, potentially misleading aspects of the data. Several aspects of a qualitative report indicate how extensively the investigators collected data: the number of observations, interviews, or documents; the duration of the observations; the duration of the study period; the diversity of units of analysis and data collection techniques; the number of investigators involved in data collection and analysis; and the degree of investigators’ involvement in data collection and analysis notes.22–,25 Taping and transcribing interviews (or other dialogue) is often desirable, but not necessary for all qualitative studies.

WERE THE DATA ANALYSED APPROPRIATELY?

Qualitative researchers often begin with a general exploratory question and preliminary concepts. They then collect relevant data, observe patterns in the data, organise these into a conceptual framework, and resume data collection to both explore and challenge their developing conceptualisations. This cycle may be repeated several times. The iterations among data collection and data interpretation continue until the analysis is well developed and further observations yield redundant, minimal, or no new information to further challenge or elaborate the conceptual framework or indepth descriptions of the phenomenon (a point often referred to as saturation26 or informational redundancy27). This “analysis stopping” criterion is so basic to qualitative analysis that authors seldom declare that they have reached this point; they assume readers will understand.

In the course of analysis, key findings may also be corroborated using several information sources, a process called data triangulation. Triangulation is a metaphor and does not mean literally that 3 or more sources are required. The appropriate number of sources depends on the importance of the findings, their implications for theory, and the investigators’ confidence in their validity. Because no 2 qualitative data sources will generate exactly the same interpretation, much of the art of qualitative interpretation involves exploring why and how different information sources yield slightly different results.28 Readers may encounter several useful triangulation techniques for validating qualitative data and their interpretation in analysis.29–,30 Investigator triangulation requires that >1 investigator collect and analyse the data, such that the findings emerge through consensus between or among investigators. This is typically accomplished by an investigative team. Inclusion of team members from different disciplines helps to prevent personal or disciplinary biases of a single researcher from excessively influencing the findings. Theory triangulation is a process whereby emergent findings are examined in relation to existing social science theories. 29,31 It is conventional for authors to report how their qualitative findings relate to prevailing social theory, although some qualitative researchers suggest that such theories should not be used to guide the research design or analysis.

Some researchers seek clarification and further explanation of their developing analytic framework from study participants, a step known as member checking. Most commonly, researchers specify that member checking was done to inquire whether participants’ viewpoints were faithfully interpreted, to determine whether there are gross errors of fact, and to ascertain whether the account makes sense to participants with different perspectives.

Some qualitative research reports describe the use of qualitative analysis software packages.32–,34 Readers should not equate the use of computers with analytic rigour. Such software is merely a data management tool for efficiently storing, organising, and retrieving qualitative data. These programs do not perform analyses. The investigators do the analysis as they create the keywords, categories, and logical relations used to organise and interpret the electronic data. The soundness of qualitative study findings depend on investigator judgments, which cannot, as yet, be programmed into software packages.

We indicated earlier that qualitative data collection must be comprehensive (ie, adequate in its breadth and depth) to yield a meaningful description. The closely related criterion for judging whether data were analysed appropriately is whether this comprehensiveness was determined in part by the research findings, with the aims of challenging, elaborating, and corroborating the findings. This is most apparent when researchers state that they alternated between data collection and analysis, collected data with the purpose of elucidating the “analysis in progress”, collected data until analytic saturation or redundancy was reached, or triangulated findings using any of the methods mentioned.

What are the findings?

IS THE DESCRIPTION OF FINDINGS THOROUGH?

Qualitative researchers are challenged to make sense of massive amounts of data and transform their understandings to a written form. The written report is often a barrier to qualitative research use because of its lack of clarity and relevance, except to a limited audience.35 Sandelowski describes the challenges facing authors, as they make decisions in balancing description (the facts of the cases observed) with analysis (the breakdown and recombining of data) and interpretation (the new meanings created from this process).35

Good research often involves “messiness”, raising as many questions as it purports to answer. Holliday describes the appropriate role of “cautious detachment” in qualitative research.16 The “truths” of qualitative research are relative to the research setting. Therefore, it is important that authors not overstep the interpretive boundaries of their study by making it seem as if all their questions were answered with certainty and without raising additional questions. A comparison of the findings and discussion sections of a study report is helpful for judging whether authors are truthful to the data and the local context of a given qualitative study.