Jochen Gläser, Stefan Lange, Grit Laudel, and Uwe Schimank
- Published in print:
- 2010
- Published Online:
- September 2010
- ISBN:
- 9780199590193
- eISBN:
- 9780191723445
- Item type:
- chapter
- Publisher:
- Oxford University Press
- DOI:
- 10.1093/acprof:oso/9780199590193.003.0005
- Subject:
- Business and Management, Public Management, Knowledge Management
This chapter suggests how the different evaluation techniques used in research evaluation systems (RES) affect their informational yield for the exercise of authority by university management. It ...
More
This chapter suggests how the different evaluation techniques used in research evaluation systems (RES) affect their informational yield for the exercise of authority by university management. It compared three peer review-based and two indicator-based systems according to their informational yield and to the use to which the information is put by university management, and uncovered clear differences between the informational yields of varied RES for universities and between the ensuing usability of information for different purposes. The clear superiority of peer reviews was demonstrated by the range of changes it enabled in the German state of Lower Saxony, as well as by the distortions resulting from the internal use of quantitative indicators in Australia. Further indirect confirmation stems from the widespread ad hoc use of peer reviews in structural decisions, and decisions about promotions to professorial levels in Australia.Less
This chapter suggests how the different evaluation techniques used in research evaluation systems (RES) affect their informational yield for the exercise of authority by university management. It compared three peer review-based and two indicator-based systems according to their informational yield and to the use to which the information is put by university management, and uncovered clear differences between the informational yields of varied RES for universities and between the ensuing usability of information for different purposes. The clear superiority of peer reviews was demonstrated by the range of changes it enabled in the German state of Lower Saxony, as well as by the distortions resulting from the internal use of quantitative indicators in Australia. Further indirect confirmation stems from the widespread ad hoc use of peer reviews in structural decisions, and decisions about promotions to professorial levels in Australia.
Thanh V. Tran
- Published in print:
- 2009
- Published Online:
- May 2009
- ISBN:
- 9780195325089
- eISBN:
- 9780199864515
- Item type:
- chapter
- Publisher:
- Oxford University Press
- DOI:
- 10.1093/acprof:oso/9780195325089.003.0002
- Subject:
- Social Work, Research and Evaluation
A research instrument is defined as a systematic and standardized tool for data collection. It includes all types of research questionnaires and standardized scales. There are three ways in ...
More
A research instrument is defined as a systematic and standardized tool for data collection. It includes all types of research questionnaires and standardized scales. There are three ways in cross-cultural research instrument development: adopting an existing instrument, adapting or modifying an existing instrument, and developing a new instrument. To develop a cross-culturally valid questionnaire or instrument, the concepts or constructs selected for the investigation must be clearly defined and bear the same meanings across the selected cultural groups. No good questionnaire can be developed without clear definitions. This is a matter of utmost importance for all levels of cultural comparative research and evaluation, whether it is a gender or racial/ethnic comparison within one society or across nations. Chapter 2 describes the process of cross-cultural instrument development, from formulating the research aims to the assessments of cross-cultural measurement properties.Less
A research instrument is defined as a systematic and standardized tool for data collection. It includes all types of research questionnaires and standardized scales. There are three ways in cross-cultural research instrument development: adopting an existing instrument, adapting or modifying an existing instrument, and developing a new instrument. To develop a cross-culturally valid questionnaire or instrument, the concepts or constructs selected for the investigation must be clearly defined and bear the same meanings across the selected cultural groups. No good questionnaire can be developed without clear definitions. This is a matter of utmost importance for all levels of cultural comparative research and evaluation, whether it is a gender or racial/ethnic comparison within one society or across nations. Chapter 2 describes the process of cross-cultural instrument development, from formulating the research aims to the assessments of cross-cultural measurement properties.
Phyllis Solomon, Mary M. Cavanaugh, and Jeffrey Draine
- Published in print:
- 2009
- Published Online:
- May 2009
- ISBN:
- 9780195333190
- eISBN:
- 9780199864317
- Item type:
- book
- Publisher:
- Oxford University Press
- DOI:
- 10.1093/acprof:oso/9780195333190.001.0001
- Subject:
- Social Work, Research and Evaluation
Randomized controlled trials (RCTs) have been considered a gold standard for health and social service research for generations of professionals. However, even as methods have developed to ...
More
Randomized controlled trials (RCTs) have been considered a gold standard for health and social service research for generations of professionals. However, even as methods have developed to accommodate a large number of professional perspectives and fields of intervention, few have adequately addressed the complex nature of RCTs conducted in community settings. In this book, Drs. Solomon, Cavanaugh, and Draine draw on their extensive experience conducting randomized controlled trials to compile a practical and accessible guide to RCTs in community-based practice settings. While providing a detailed, common-sense manual, the authors address numerous design and implementation challenges that are unique to practice settings, which are less-controlled environments than the typical clinic or consultation room. Such issues include: community and agency buy-in to support collaboration, addressing confounds to internal and external validity, and fidelity with complex interventions. These challenges are addressed through a mix of qualitative and quantitative methods that have supported RCT research in community-based settings. This pragmatic guide provides a thorough review of the basic ingredients for working through each step of the RCT process. It offers encouragement and support to enter this richly rewarding and challenging research area.Less
Randomized controlled trials (RCTs) have been considered a gold standard for health and social service research for generations of professionals. However, even as methods have developed to accommodate a large number of professional perspectives and fields of intervention, few have adequately addressed the complex nature of RCTs conducted in community settings. In this book, Drs. Solomon, Cavanaugh, and Draine draw on their extensive experience conducting randomized controlled trials to compile a practical and accessible guide to RCTs in community-based practice settings. While providing a detailed, common-sense manual, the authors address numerous design and implementation challenges that are unique to practice settings, which are less-controlled environments than the typical clinic or consultation room. Such issues include: community and agency buy-in to support collaboration, addressing confounds to internal and external validity, and fidelity with complex interventions. These challenges are addressed through a mix of qualitative and quantitative methods that have supported RCT research in community-based settings. This pragmatic guide provides a thorough review of the basic ingredients for working through each step of the RCT process. It offers encouragement and support to enter this richly rewarding and challenging research area.
Dominic McVey, Adam Crosier, and Alex Christopoulos
- Published in print:
- 2009
- Published Online:
- February 2010
- ISBN:
- 9780199550692
- eISBN:
- 9780191720413
- Item type:
- chapter
- Publisher:
- Oxford University Press
- DOI:
- 10.1093/acprof:oso/9780199550692.003.14
- Subject:
- Public Health and Epidemiology, Public Health, Epidemiology
This chapter describes the common approaches to evaluation, which is the fourth step in the Total Process Planning (TPP) framework. As well as setting out the theory behind designing and conducting ...
More
This chapter describes the common approaches to evaluation, which is the fourth step in the Total Process Planning (TPP) framework. As well as setting out the theory behind designing and conducting evaluation research, it provides case studies that show how evaluation works in practice. By the time the evaluation phase is reached, you should have completed the scoping report, and developed and implemented your marketing strategy. However, evaluation is not a distinct stage that only needs to be addressed when you have implemented your intervention. It must be considered from the scoping stage onwards.Less
This chapter describes the common approaches to evaluation, which is the fourth step in the Total Process Planning (TPP) framework. As well as setting out the theory behind designing and conducting evaluation research, it provides case studies that show how evaluation works in practice. By the time the evaluation phase is reached, you should have completed the scoping report, and developed and implemented your marketing strategy. However, evaluation is not a distinct stage that only needs to be addressed when you have implemented your intervention. It must be considered from the scoping stage onwards.
Bruce A. Thyer
- Published in print:
- 2012
- Published Online:
- May 2012
- ISBN:
- 9780195387384
- eISBN:
- 9780199932085
- Item type:
- book
- Publisher:
- Oxford University Press
- DOI:
- 10.1093/acprof:oso/9780195387384.001.0001
- Subject:
- Social Work, Research and Evaluation
Quasi-experimental research designs are the most widely used research approach employed to evaluate the outcomes of social work programs and policies. This new volume describes the logic, design, and ...
More
Quasi-experimental research designs are the most widely used research approach employed to evaluate the outcomes of social work programs and policies. This new volume describes the logic, design, and conduct of the range of such designs, encompassing pre-experiments, quasi-experiments making use of a control or comparison group, and time-series designs. An introductory chapter describes the valuable role these types of studies have played in social work, going back to the 1930s, and continuing to the present. Subsequent chapters describe the major features of individual quasi-experimental designs, the types of questions they are capable of answering, and their strengths and limitations. Each discussion of these designs presented in the abstract is subsequently illustrated with descriptions of real examples of their use as published in the social work literature and related fields. By linking the discussion of quasi-experimental designs in the abstract to actual applications to evaluate the outcomes of social services, the usefulness and vitality of these research methods comes alive to the reader. While this volume could be used as a research textbook, it will also have great value to practitioners seeking a greater conceptual understanding of the quasi-experimental studies they frequently read about in the social work literature. Human service professionals planning to undertake a program evaluation of their own agency's services will find this book of immense help in understanding the steps and actions needed to adopt a quasi-experimental strategy. It is usually the case that ethical and pragmatic considerations preclude the use of randomly assigning social work clients to experimental and comparative treatment conditions, and in such instances, the practicality of employing a quasi-experimental method becomes an excellent alternative.Less
Quasi-experimental research designs are the most widely used research approach employed to evaluate the outcomes of social work programs and policies. This new volume describes the logic, design, and conduct of the range of such designs, encompassing pre-experiments, quasi-experiments making use of a control or comparison group, and time-series designs. An introductory chapter describes the valuable role these types of studies have played in social work, going back to the 1930s, and continuing to the present. Subsequent chapters describe the major features of individual quasi-experimental designs, the types of questions they are capable of answering, and their strengths and limitations. Each discussion of these designs presented in the abstract is subsequently illustrated with descriptions of real examples of their use as published in the social work literature and related fields. By linking the discussion of quasi-experimental designs in the abstract to actual applications to evaluate the outcomes of social services, the usefulness and vitality of these research methods comes alive to the reader. While this volume could be used as a research textbook, it will also have great value to practitioners seeking a greater conceptual understanding of the quasi-experimental studies they frequently read about in the social work literature. Human service professionals planning to undertake a program evaluation of their own agency's services will find this book of immense help in understanding the steps and actions needed to adopt a quasi-experimental strategy. It is usually the case that ethical and pragmatic considerations preclude the use of randomly assigning social work clients to experimental and comparative treatment conditions, and in such instances, the practicality of employing a quasi-experimental method becomes an excellent alternative.
MARGARET Robbins
- Published in print:
- 1998
- Published Online:
- November 2011
- ISBN:
- 9780192626219
- eISBN:
- 9780191730016
- Item type:
- chapter
- Publisher:
- Oxford University Press
- DOI:
- 10.1093/acprof:oso/9780192626219.003.0006
- Subject:
- Palliative Care, Palliative Medicine Research, Patient Care and End-of-Life Decision Making
This chapter highlights the gaps in the evidence base of palliative care and attempts to provide some pointers on the future direction of research in the evaluation of palliative care. It discusses ...
More
This chapter highlights the gaps in the evidence base of palliative care and attempts to provide some pointers on the future direction of research in the evaluation of palliative care. It discusses the strengths and weakness of evaluation research and proposes different ways to enhance the effectiveness of evaluation studies. The chapter also explores the policy process concerning palliative care planning and provision.Less
This chapter highlights the gaps in the evidence base of palliative care and attempts to provide some pointers on the future direction of research in the evaluation of palliative care. It discusses the strengths and weakness of evaluation research and proposes different ways to enhance the effectiveness of evaluation studies. The chapter also explores the policy process concerning palliative care planning and provision.
Terry Moore
- Published in print:
- 2010
- Published Online:
- May 2010
- ISBN:
- 9780195321302
- eISBN:
- 9780199777457
- Item type:
- chapter
- Publisher:
- Oxford University Press
- DOI:
- 10.1093/acprof:oso/9780195321302.003.0008
- Subject:
- Social Work, Children and Families, Communities and Organizations
The lengthy observational period required for many experimental and observational studies in child welfare has lead to greater interest in formative evaluations for guiding quality improvement. This ...
More
The lengthy observational period required for many experimental and observational studies in child welfare has lead to greater interest in formative evaluations for guiding quality improvement. This chapter offers practical advice for opening up the proverbial “black box” of evaluation research and for developing data-tracking systems for continuous monitoring of service implementation, program outputs, and longer-term outcomes. It discusses the importance of fostering a results-oriented organizational culture that communicates the attitudes, behaviors, and values of results-oriented accountability (ROA).Less
The lengthy observational period required for many experimental and observational studies in child welfare has lead to greater interest in formative evaluations for guiding quality improvement. This chapter offers practical advice for opening up the proverbial “black box” of evaluation research and for developing data-tracking systems for continuous monitoring of service implementation, program outputs, and longer-term outcomes. It discusses the importance of fostering a results-oriented organizational culture that communicates the attitudes, behaviors, and values of results-oriented accountability (ROA).
James Ptacek
- Published in print:
- 2009
- Published Online:
- February 2010
- ISBN:
- 9780195335484
- eISBN:
- 9780199864331
- Item type:
- chapter
- Publisher:
- Oxford University Press
- DOI:
- 10.1093/acprof:oso/9780195335484.003.0013
- Subject:
- Social Work, Children and Families, Crime and Justice
In this concluding chapter, the commonalities and differences among the books’ contributors are discussed. In particular, the various meanings given to “justice” and “restoration” by the authors are ...
More
In this concluding chapter, the commonalities and differences among the books’ contributors are discussed. In particular, the various meanings given to “justice” and “restoration” by the authors are considered. If justice is seen as, in part, an experience, the question is raised as to where, in the practices described in the book, justice is thought to happen. The chapter ends with recommendations concerning new justice practices that address violence against women, with an emphasis on screening for abuse, advocate-researcher collaborations, and evaluation research.Less
In this concluding chapter, the commonalities and differences among the books’ contributors are discussed. In particular, the various meanings given to “justice” and “restoration” by the authors are considered. If justice is seen as, in part, an experience, the question is raised as to where, in the practices described in the book, justice is thought to happen. The chapter ends with recommendations concerning new justice practices that address violence against women, with an emphasis on screening for abuse, advocate-researcher collaborations, and evaluation research.
Neil J. Smelser and John S. Reed
- Published in print:
- 2012
- Published Online:
- May 2013
- ISBN:
- 9780520273566
- eISBN:
- 9780520954144
- Item type:
- chapter
- Publisher:
- University of California Press
- DOI:
- 10.1525/california/9780520273566.003.0009
- Subject:
- Sociology, Social Research and Statistics
The aim of this chapter is to identify the major methods of research in the social sciences and to examine the relative strengths and weaknesses of each method with respect to their relevance and ...
More
The aim of this chapter is to identify the major methods of research in the social sciences and to examine the relative strengths and weaknesses of each method with respect to their relevance and usefulness in the arenas decision making, social policy, and social problems. The different research methods are classified as laboratory-experimental, evaluation research, statistic methods and survey research, comparative-historical analysis, case studies, counterfactual reasoning and mental experiments. These methods are treated as variations—but identical in aim—in the process of varying and holding relevant variables constant in order to arrive at adequate causal explanations. The authors assess strengths and weaknesses of each as they are applied to “real” situations in the “real” world.Less
The aim of this chapter is to identify the major methods of research in the social sciences and to examine the relative strengths and weaknesses of each method with respect to their relevance and usefulness in the arenas decision making, social policy, and social problems. The different research methods are classified as laboratory-experimental, evaluation research, statistic methods and survey research, comparative-historical analysis, case studies, counterfactual reasoning and mental experiments. These methods are treated as variations—but identical in aim—in the process of varying and holding relevant variables constant in order to arrive at adequate causal explanations. The authors assess strengths and weaknesses of each as they are applied to “real” situations in the “real” world.
Corey S. Shdaimah, Roland W. Stahl, and Sanford F. Schram
- Published in print:
- 2011
- Published Online:
- November 2015
- ISBN:
- 9780231151795
- eISBN:
- 9780231525367
- Item type:
- chapter
- Publisher:
- Columbia University Press
- DOI:
- 10.7312/columbia/9780231151795.003.0002
- Subject:
- Social Work, Research and Evaluation
This chapter discusses evaluation research and how it can be informed by the theory of change research. More specifically, it considers how community-based practice must increasingly document ...
More
This chapter discusses evaluation research and how it can be informed by the theory of change research. More specifically, it considers how community-based practice must increasingly document effectiveness through research that systematically evaluates whether community-based programs and services produce intended results, especially when funders demand this type of accountability. Although accountability is undoubtedly a legitimate goal, the chapter argues that the type of evaluation research that is demanded in the name of accountability often leaves a great deal to be desired. It also highlights the problems with documenting effectiveness via evaluation research as well as the criticisms against measurement associated with performance management, such as disconnecting social work researchers from their community partners to the point of suspicion and distrust instead of promoting the synergy and increased effectiveness that can come from combining forces to revise and improve practice.Less
This chapter discusses evaluation research and how it can be informed by the theory of change research. More specifically, it considers how community-based practice must increasingly document effectiveness through research that systematically evaluates whether community-based programs and services produce intended results, especially when funders demand this type of accountability. Although accountability is undoubtedly a legitimate goal, the chapter argues that the type of evaluation research that is demanded in the name of accountability often leaves a great deal to be desired. It also highlights the problems with documenting effectiveness via evaluation research as well as the criticisms against measurement associated with performance management, such as disconnecting social work researchers from their community partners to the point of suspicion and distrust instead of promoting the synergy and increased effectiveness that can come from combining forces to revise and improve practice.
Kaye Wellings and Wendy Macdowall
- Published in print:
- 2004
- Published Online:
- September 2009
- ISBN:
- 9780198528807
- eISBN:
- 9780191723964
- Item type:
- chapter
- Publisher:
- Oxford University Press
- DOI:
- 10.1093/acprof:oso/9780198528807.003.0011
- Subject:
- Public Health and Epidemiology, Public Health, Epidemiology
Broad spectrum interventions aimed at reaching the general population make use of mass communication approaches such as TV, radio, press, billboard posters, and leaflets. This chapter examines the ...
More
Broad spectrum interventions aimed at reaching the general population make use of mass communication approaches such as TV, radio, press, billboard posters, and leaflets. This chapter examines the difficulties inherent in evaluating mass media campaigns. Topics discussed include the scope of interventions, evaluation research, process evaluation, outcome evaluation, selection of indicators/outcome measures, and measuring unintended consequences.Less
Broad spectrum interventions aimed at reaching the general population make use of mass communication approaches such as TV, radio, press, billboard posters, and leaflets. This chapter examines the difficulties inherent in evaluating mass media campaigns. Topics discussed include the scope of interventions, evaluation research, process evaluation, outcome evaluation, selection of indicators/outcome measures, and measuring unintended consequences.
Mats Alvesson, Yiannis Gabriel, and Roland Paulsen
- Published in print:
- 2017
- Published Online:
- June 2017
- ISBN:
- 9780198787099
- eISBN:
- 9780191829161
- Item type:
- chapter
- Publisher:
- Oxford University Press
- DOI:
- 10.1093/oso/9780198787099.003.0008
- Subject:
- Business and Management, Organization Studies
This chapter examines how wider social forces, including globalization, neo-liberal economics, and widening participation in higher education shape government policies, and how much space these allow ...
More
This chapter examines how wider social forces, including globalization, neo-liberal economics, and widening participation in higher education shape government policies, and how much space these allow for academics to pursue meaningful and socially useful research. It assesses how various research evaluation schemes can be made more effective and meaningful, and how they may encourage more relevant research in the social sciences. The chapter addresses the use of bibliometrics (citations indices) and measurements aimed at assessing the impact of academic research. It also examines how teaching may be restored as in its rightful place as a core meaningful activity for social science academics. Teaching must once again anchor the identities and practices of academics, whether they see themselves as research-active or inactive, as long as they seek to maintain active scholarly identities.Less
This chapter examines how wider social forces, including globalization, neo-liberal economics, and widening participation in higher education shape government policies, and how much space these allow for academics to pursue meaningful and socially useful research. It assesses how various research evaluation schemes can be made more effective and meaningful, and how they may encourage more relevant research in the social sciences. The chapter addresses the use of bibliometrics (citations indices) and measurements aimed at assessing the impact of academic research. It also examines how teaching may be restored as in its rightful place as a core meaningful activity for social science academics. Teaching must once again anchor the identities and practices of academics, whether they see themselves as research-active or inactive, as long as they seek to maintain active scholarly identities.
Michael Rutter
- Published in print:
- 2007
- Published Online:
- March 2012
- ISBN:
- 9781861349507
- eISBN:
- 9781447303909
- Item type:
- chapter
- Publisher:
- Policy Press
- DOI:
- 10.1332/policypress/9781861349507.003.0011
- Subject:
- Sociology, Marriage and the Family
This chapter examines the need for a new initiative, the evidence for this, the content of the provision, the design of the programme, and the research evaluation and design, and concludes with an ...
More
This chapter examines the need for a new initiative, the evidence for this, the content of the provision, the design of the programme, and the research evaluation and design, and concludes with an overview. It brings out positive elements of the initiative, as well as making critical comments. The chapter also focuses on effects on child and family functioning rather than on the longer-term and more indirect effects on child poverty. It questions whether the government has learnt any lessons from the experience of the implementation and evaluation of SSLPs.Less
This chapter examines the need for a new initiative, the evidence for this, the content of the provision, the design of the programme, and the research evaluation and design, and concludes with an overview. It brings out positive elements of the initiative, as well as making critical comments. The chapter also focuses on effects on child and family functioning rather than on the longer-term and more indirect effects on child poverty. It questions whether the government has learnt any lessons from the experience of the implementation and evaluation of SSLPs.
Carol M. Ashton and Nelda P. Wray
- Published in print:
- 2013
- Published Online:
- September 2013
- ISBN:
- 9780199968565
- eISBN:
- 9780199346080
- Item type:
- chapter
- Publisher:
- Oxford University Press
- DOI:
- 10.1093/acprof:oso/9780199968565.003.0003
- Subject:
- Public Health and Epidemiology, Public Health
The principles of design, conduct, analysis and reporting of clinical trials, including trials conducted for regulatory purposes, are sufficiently advanced that drugs and other therapeutic ...
More
The principles of design, conduct, analysis and reporting of clinical trials, including trials conducted for regulatory purposes, are sufficiently advanced that drugs and other therapeutic interventions can be put to fair tests that provide valid evidence of efficacy and safety. However, clinical research—especially research that can make a product a winner or a loser—is not a value-free human endeavor. The rosiglitazone story exemplifies how medical product manufacturers can manipulate regulatory evidence by designing trials to ensure certain answers are—or are not—obtained, how companies can try to intimidate scientists whose research yields unfavorable evidence about a product, and how ambiguous can be the evidence from clinical trials. Federally-mandated comparative effectiveness research does not generate evidence used in product regulation, but it generates evidence that has market implications. It therefore will always be at risk for the same kinds of manipulation observed in the rosiglitazone saga.Less
The principles of design, conduct, analysis and reporting of clinical trials, including trials conducted for regulatory purposes, are sufficiently advanced that drugs and other therapeutic interventions can be put to fair tests that provide valid evidence of efficacy and safety. However, clinical research—especially research that can make a product a winner or a loser—is not a value-free human endeavor. The rosiglitazone story exemplifies how medical product manufacturers can manipulate regulatory evidence by designing trials to ensure certain answers are—or are not—obtained, how companies can try to intimidate scientists whose research yields unfavorable evidence about a product, and how ambiguous can be the evidence from clinical trials. Federally-mandated comparative effectiveness research does not generate evidence used in product regulation, but it generates evidence that has market implications. It therefore will always be at risk for the same kinds of manipulation observed in the rosiglitazone saga.
Patricia Leavy
- Published in print:
- 2011
- Published Online:
- March 2015
- ISBN:
- 9780195395099
- eISBN:
- 9780190255756
- Item type:
- chapter
- Publisher:
- Oxford University Press
- DOI:
- 10.1093/acprof:osobl/9780195395099.003.0005
- Subject:
- Psychology, Clinical Psychology
This chapter reviews a range of concepts appropriate for evaluating oral history research. It presents strategies researchers can employ in their own research in order to address each dimension on ...
More
This chapter reviews a range of concepts appropriate for evaluating oral history research. It presents strategies researchers can employ in their own research in order to address each dimension on which oral history may be judged. It discusses concepts such as explicitness, thoroughness and congruence, ethical practice, and validity and related concepts.Less
This chapter reviews a range of concepts appropriate for evaluating oral history research. It presents strategies researchers can employ in their own research in order to address each dimension on which oral history may be judged. It discusses concepts such as explicitness, thoroughness and congruence, ethical practice, and validity and related concepts.
