Course Description (04:33)
- Help decision makers make well-informed decisions on policies, programs, and projects by using research in policy development and implementation.
- Advocates a more rational, rigorous, systematic approach to research to adopt a broader understanding.
- The purpose is to improve public management and policy-making by grounding decision making in evidence.
- Evidence‐Based Management enhances the overall quality of organizational decisions and practices through reflective use of relevant and best available scientific evidence.
Combines conscientious, judicious use of best evidence with individual expertise; ethics; valid, reliable facts; and consideration of impact on stakeholders.
- Do things that help people, never do things that hurt people.
Success is enhanced by quality connections among practitioners, management, educators, and scholars.
- It is highly encouraged to build connections.
- Evidence-Based Practices and Evidence-Based Management are used interchangeably throughout the course.
- “How empty is theory in the presence of fact!” Mark Twain, A Connecticut Yankee in King Arthur's Court
- “Insanity: doing the same thing over and over again and expecting different results.” Albert Einstein
Contention and confusion on what constitutes scholarly evidence:
Evidence should come from a variety of sources, aka The Goldilocks Theory
- Expert opinions
- High quality, vs best available, vs good enough, vs only available vs compliant
- Only evaluations based on rigorous experimentation, research, and trials are considered evidence.
- A pragmatic middle ground suggesting some methods are better suited than others.
- No evidentiary method is without serious flaws, variety of methods must be used to triangulate.
- Evidence should come from a variety of sources, aka The Goldilocks Theory
In the real world of management and policy-making, most practitioners treat evidence as only one of several factors taken into account in decision making.
Points to be made:
- Theory is theory, not fact.
- With theory, we are always dealing with probability or a percentage chance
- Theory integration vs stand-alone theories increases explanatory power
- Never dismiss common sense (Practitioner based)
- Generalizability – is the theory /result capable of applying to or for a wider audience than the research group in question.
- No such thing as perfect research
- Theories are not created in a vacuum
Theory and theorists are not pure of heart and mind
- Evidence-based policy development is applicable to all disciplines whether it is for and/or from a soft or hard science.
Evidence-Based Management (16:03)
An attitude of mind that:
- Thinks in terms of evidence for decisions and about the nature of the evidence;
Asks questions such as:
- What is happening?
- How is it happening?
- What are the consequences?
- Is aware of the potential limitations of the different answers; and
- Is interested in research to try to find the answers or at least to reduce the ignorance
Evidence-Based Management is rooted from Evidence-Based Medicine
- What the doctor says isn’t the gospel – it works to get a second opinion.
- Evidence‐based medicine “the systematic identification, appraisal and synthesis of clinical studies, particularly randomized clinical trials” was advanced to replace the art of medicine and physician practices based on personal journeys but contradicted by scientific evidence.
The struggle to change medical practice based on research evidence has a long history, with valuable implications for public safety agencies (PSAs).
- In the 1840s, Ignaz Semmelweiss found evidence that maternal death in childbirth could be reduced if doctors washed their hands before delivering babies. He then tried to apply this research to medical practice in Vienna, which led to his being driven out of town by his boss, the chief obstetrician. Hundreds of thousands of women died because the profession refused to comply with his evidence‐based guidelines for some forty years.
- The story shows the important distinction between merely doing research and attempting to apply research to redirect professional practices. One way to describe people who try to apply research is the role of “evidence cop.”
New evidence shows that doctors resist changing practices based on new research just as much as PSAs do, if not more so.
Those lessons come from a new strategy called “evidence‐based medicine,” “widely hailed as the long‐sought link between research and practice” to solve problems like the following:
- An estimated 85 percent of medical practices remain untested by research evidence.
- Most doctors rarely read the 2,500 medical journals available, and instead, base their practice on local custom.
NIH convenes advisory boards to issue to physicians’ recommendations that are based on intensive reviews of research evidence that receive extensive publicity, and are reinforced by mailings of the guideline summaries to some one hundred thousand doctors.
Doctors rarely change their practices in response to publication of these guidelines.
- Three years after research found that heart attack patients treated with calcium antagonists were more likely to die, doctors still prescribed this dangerous drug to one‐third of heart attack patients.
- Eight years after antibiotics were shown to cure ulcers, 90 percent of ulcer patients remained untreated by antibiotics.
- Doctors rarely change their practices in response to publication of these guidelines.
- The basic premise of evidence‐based practice is that we are all entitled to our own opinions, but not to our own facts. Yet left alone to practice individually, practitioners do come up with their own “facts,” which often turn out to be wrong.
- Those lessons come from a new strategy called “evidence‐based medicine,” “widely hailed as the long‐sought link between research and practice” to solve problems like the following:
- Difference between Academics and Practitioners
Orientation of research
Focus of research
Ad hoc, ambiguous
Narrow and institutionalized
Wide and eclectic
Criteria of goodness
- Practitioners want useful information.
Why does EBP matter? (23:32)
We need evidence to:
- Understand the policy environment and how it’s changing.
Appraise the likely effects of policy changes so we can choose between different policy options and subsequently assess their impacts.
- What we do can impact people.
- Demonstrate the links between strategic direction, intended outcomes and policy objectives, to show that there are clear lines of argument and evidence between what we are aiming for and what we are doing now.
- Determine what we need to do to meet our strategic goals and take them through to delivery.
Communicate the quality (breadth and depth) of our evidence-based to meet the open government agenda.
- It should be available for debate
Attempts to establish what evidence is useful to policymakers
- This refers to the accuracy of the evidence.
- Credible evidence relies on a strong and clear line of argument.
- Tried and tested analytical methods.
- Statistically significant findings.
- The key issue here is that evidence is timely, topical and has policy implications.
- This relates to the extent to which the evidence is accessible to policymakers; whether policymakers have access to it in a useful form and therefore the ease with which it can be translated into policy.
- Problem definition/Agenda setting
- Constructing the policy alternatives/Policy formulation
- Choice of solution/selection of preferred policy option
- Takes a lot of work
- Policy implementation and monitoring
Monitoring and Policy Evaluation
- Agenda Setting
EBP Flow Chart (26:34)
Based on medical practices
Apply principles of EBP
- Seek evidence (literature, data, local info)
- Consult health service personnel (educators, managers, policymakers)
- Don’t do it in a vacuum
- Identify need for change
- Develop proposal change
- Implement change
Evaluate extent and results of change
- You can’t evaluate what you can’t measure
- Ensure sustainability, avoid duplication and integrate with existing systems
Document and investigate the change process in an action research approach
- Take note of outlier factors and compounding variables
- Apply principles of EBP
Diagram of the flow of evidence in longer-term policy and strategy development
- Scope issue
- Ask the question
- Decide on evidence needed
- Procure, manage and carry out research
Interpret and apply new or existing evidence
- Somebody might have already had the same problem you have
- Monitor and evaluate the policy once implemented
Factors influencing policymaking in Government
- At the core of decision making must be evidence
Experience, Expertise, and Judgement
- Human and intellectual capital
- Have the needed resources to do what you need to do
- Lean left or right politically
- What are your pre-conceived notions
Habit and Tradition
- Often defy rational explanation in the twenty‐first century;
Lobbyist, Pressure Groups, and Consultants
- Think tanks
- Opinion leaders
Pragmatics and Contingencies
- Capacities of institutions
- Unanticipated contingencies
An alternative way of categorizing the factors which policy and practice depend on.
- As much knowledge as we can have
- Be careful of self‐interests
- Political orientation
- Value assigned to research is less than prevailing thought or opinion.
- Evidence, therefore, has a tough role of play if it is to gain wider credibility with decision makers.
- Time constraints and the resultant pressure should feature as a stand‐alone factor.
Things Affecting EBP (31:39)
What limits EBP (5S)
- Time issue
- Needs to get things done now
- Stop-Gap measures/Bad-aid fixes
- Know who to turn to for advice
- Perception guides political decisions
- Scientific ignorance
9 Key Characteristics which policy-making should aspire to:
- We don’t always have the answers but someone else might
Innovative and creative
- Challenges the status quo
- Work across boundaries
- Keep up to date
- Better to learn from someone else’s mistakes
Encouraging better use of evidence in policymaking
- Increasing the pull for evidence
- Require the publication of the evidence-based for policy decisions
Require departmental spending bids to provide a supporting evidence base
- Don’t just ask for budget, show how the money spent will work and benefit organization
Submit governmental analysis (such as forecasting models) to external expert scrutiny
- Encouraged to work with the academic community
- Provide open access to information – leading to more informed citizens and pressure groups
- Facilitating better evidence use
Encourage better collaboration across internal analytical services
- Co‐locate policymakers and internal analysts
- Integrate analytical staff at all stages of the policy development process
- Link R&D strategies to departmental business plans
- Cast external researchers more as partners than as contractors
- Train staff in evidence use
Types of economic analysis used in economic evaluation
There’s a finite amount of money, cannot be wasted
- Cost-effectiveness analysis
- Cost-benefit analysis
- Cost-utility analysis
- There’s a finite amount of money, cannot be wasted
Practice of systematic reviews
Developing the initial question
- What is the problem being addressed?
Clarifying which studies are relevant
- Someone needs to do research
- Identifying studies
- Checking that the studies found meet the selection criteria
- Further coding of studies for synthesis
- Quality and relevance appraisal
- Recommendations and guidance
- Developing the initial question
Document analysis with visual artifacts
- Crime data
- Fire data
- Blogs, journals
- Incident reports
- Training Materials
- Public records
Disconnection, mistrust and poor understanding between the worlds of ideas/research and action/practice
- Fix disconnections
- Reduce mistrust
- Improve understanding
A static view of academic research
- Opt for a dynamic view
- Develop relationship with those who can help and work with you
Few skills or incentives to do applied research
- Disconnection, mistrust and poor understanding between the worlds of ideas/research and action/practice
Evidence-Based Policy System (39:30)
What would the perfect evidence‐based policy system look like?
- Policy advisors have the capacity to act and the competencies to understand the choices available.
Works beyond the electoral cycle and focuses on long‐term issues .
- Not dependent on who’s holding the power
- A system that utilizes existing capacity.
- Proactive to changes in the field of action.
- With room for experimentation.
Where innovation is incentivized.
- Encourage and reward people to do so.
- Where the capacity to speak truth to power exists.
- Clear accountabilities.
- Policy and evidence are effectively integrated.
- Information systems allow for the effective flow of information.
- Evidence is freely debated and shared.
- Better practice is shared.
- Access to evidence and strong productive working relationships and knowledge institutions.
- Effective use of innovation intermediaries.
- Demand and supply-side incentives to engage in evidence‐based decision-making.
Evidence-based managers work through this sequence
- Collect evidence that a problem exists
- Diagnose the root causes of the problem
- Prescribe the best possible solution
- Implement the solution
- Check for evidence that the solution worked, and, if it did not, start again at step 1 by revisiting all available evidence.
Evidence, Statistics and Darn Lies
Evidence itself, regardless of its quality or veracity, will never be enough
- There is no single, absolute definition of what constitutes valid evidence
- One man’s “evidence” is another man’s “statistics and darn lies”
- In an organizational context, particularly a public management organization, political expediency can easily trump and truncate evidence‐based philosophy, policy, and practice
- Evidence itself, regardless of its quality or veracity, will never be enough
Three Broad Interrelated Activities: An Evidence‐Informed Approach
- Deciding and responding
- Developing evidence
- Communication and networking
Utilizing Research to Strengthen Legislation (42:26)
Tennessee Bureau of Investigation (TBI) conducted research to discover just how prevalent human trafficking was in the state.
- TBI completed a research study entitled Tennessee Human Sex Trafficking and Its Impact on Children and Youth.
- TBI established a full‐time statewide human trafficking unit
Since 2011, the Tennessee General Assembly has passed 39 pieces of legislation and had positive results
- Strengthened the laws to protect survivors of human trafficking
- Prosecuting offenders by employing the new human trafficking laws
Tallahassee Police Department completed a research project entitled Pre ‐Arrest Diversion Program
- Diverts first‐time misdemeanor offenders directly to a local behavioral health agency for intervention services.
- Intervention services provided by the behavioral health agency address criminogenic and substance abuse behaviors to reduce recidivism and improve public safety.
- The research utilized evidence‐-based intervention services that have significantly influenced recidivism rates.
- The Western Carolina University’s Department of Criminology and Criminal Justice guided the complex research effort.
The City of Redlands, CA, Police Department completed a research study entitled Translating “Near Repeat” Theory into a Geospatial Policing Strategy: A Randomized Experiment Testing a Theoretically Informed Strategy for Preventing Residential Burglary
- Randomized controlled trial to test whether quickly notifying community residents that they are at an increased risk for a burglary and providing them with burglary prevention tips reduced incidents of further burglaries in the high‐risk time period.
- This research was the first systematic test of a policing strategy designed to disrupt the near‐-repeat pattern of residential burglary.
- Impacted the community in a positive way that made community members more aware of crime in their neighborhoods and causing them to be more vigilant about locking doors and windows, to watch out for their neighbors, and to be more likely to report a burglary to the police.
- Crime Prevention Matrix from the George Mason University
- Tennessee Bureau of Investigation (TBI) conducted research to discover just how prevalent human trafficking was in the state.
If we knew what it was we were doing, it would not be called research, would it? – Albert Einstein
- It’s ok not to know, but it’s not ok to remain unknowing
- One of the greatest tragedies of life is the murder of a beautiful theory by a gang of facts – Benjamin Franklin
Statistics from the American Society of Evidence-Based Policing
- 18,000 US Police Agencies
- 900,00 law enforcement officers
- 100 B spent in the US for policing
- Unknown number of police departments using EBP