(I lost my first attempt at this essay, so I hope I don’t inadvertently submit this twice). First of all, I want to praise Silverstein and Maher for being brave enough to air their experience, warts and all. If more folks would fess up to these kinds of problems regarding research and evaluation, perhaps more would be willing to admit that the Emporer has no clothes (or is scantily clad at best). As the Evaluation Coordinator for a nonprofit organization, your article struck a very familiar chord. I guided one of the organization’s projects through the national model program process, so I am all too familiar with the ups and downs of the whole Evidence-Based Practice movement. For the most part, the EBP movement is based on principles and assumptions from basic research, including the randomized control trial design - the infamous “gold standard”. RCT designs are useful in BASIC research, when there is a high degree of control over the variables in question. In FIELD-BASED research, however, it is usually not possible to control things so tightly, thereby requiring different assumptions, models, analytic tools, etc. I find that most folks (this includes funders and EBP regulators) don’t really know the difference and continue to apply basic research tools to field based studies. For example, with randomization, one can assume a fair amount of equivalence between treatement and control groups. Without randomization, as in your study, you cannot assume this and you must use certain statistical tools to determine and adjust for these differences. These may require more sophistication than what is usually offered through university courses. Another problem I often see (as seems to be the case with your study) in both field based and basic research is that the intervention group is compared to a control group that receives no services (after school program vs. no after school program). So what you may unwittingly be comparing is whether ANY activity after school is better/worse/the same as NO activity after school. It might make more sense to compare to a third group with some other after school activity (enrichment program, sports, scouts, etc.) to really understand what might be going on. Again, many thanks for your frank discussion. The whole EBP movement, while well intentioned, has become a high stakes endeavor, leaving lots of good programs out in the cold.
Susan Carter
Mind Over Matter Consulting
Your frustration is collective. We run an arts-based program for high risk youth that includes new Canadians, for ages 12 years - 24 years. We provde holistic support, including mental health, court, and addiction support, plus all of the arts (such as music, clothing design, writing, school tutoring, drama, etc.) There is increasing pressure to “prove” that these initiatives actully work for the mainstream societal goals of increased employment, better education, and overall…the perfect “citizen” as defined by government funders. Obviously, we all need to tap into the Private Sector more and find champions that understand the reality of effective grass roots organizations because they have somehow experienced the transformation process in their own lives. Find these people and you will find your champions. You can google ihuman or Wallis Kendal and get a better picture of our process. We have good press, but very little money…lots of dedicated volunteers.
Yes, its great to have supportive local NGO’s helping you, but government partnerships are always a good idea, in my opinion. One resource (which of course depends on your religious affiliations) would be sunday school programs at larger churches, which, although not government funded, usually have many resources due to tax breaks. I wish you and New Futures all the best. I hope things have improved since 2007. -Gerry
COMMENTS
BY susancarter
ON December 20, 2007 03:39 PM
(I lost my first attempt at this essay, so I hope I don’t inadvertently submit this twice). First of all, I want to praise Silverstein and Maher for being brave enough to air their experience, warts and all. If more folks would fess up to these kinds of problems regarding research and evaluation, perhaps more would be willing to admit that the Emporer has no clothes (or is scantily clad at best). As the Evaluation Coordinator for a nonprofit organization, your article struck a very familiar chord. I guided one of the organization’s projects through the national model program process, so I am all too familiar with the ups and downs of the whole Evidence-Based Practice movement. For the most part, the EBP movement is based on principles and assumptions from basic research, including the randomized control trial design - the infamous “gold standard”. RCT designs are useful in BASIC research, when there is a high degree of control over the variables in question. In FIELD-BASED research, however, it is usually not possible to control things so tightly, thereby requiring different assumptions, models, analytic tools, etc. I find that most folks (this includes funders and EBP regulators) don’t really know the difference and continue to apply basic research tools to field based studies. For example, with randomization, one can assume a fair amount of equivalence between treatement and control groups. Without randomization, as in your study, you cannot assume this and you must use certain statistical tools to determine and adjust for these differences. These may require more sophistication than what is usually offered through university courses. Another problem I often see (as seems to be the case with your study) in both field based and basic research is that the intervention group is compared to a control group that receives no services (after school program vs. no after school program). So what you may unwittingly be comparing is whether ANY activity after school is better/worse/the same as NO activity after school. It might make more sense to compare to a third group with some other after school activity (enrichment program, sports, scouts, etc.) to really understand what might be going on. Again, many thanks for your frank discussion. The whole EBP movement, while well intentioned, has become a high stakes endeavor, leaving lots of good programs out in the cold.
Susan Carter
Mind Over Matter Consulting
BY Wallis Kendal
ON December 25, 2007 11:48 AM
Your frustration is collective. We run an arts-based program for high risk youth that includes new Canadians, for ages 12 years - 24 years. We provde holistic support, including mental health, court, and addiction support, plus all of the arts (such as music, clothing design, writing, school tutoring, drama, etc.) There is increasing pressure to “prove” that these initiatives actully work for the mainstream societal goals of increased employment, better education, and overall…the perfect “citizen” as defined by government funders. Obviously, we all need to tap into the Private Sector more and find champions that understand the reality of effective grass roots organizations because they have somehow experienced the transformation process in their own lives. Find these people and you will find your champions. You can google ihuman or Wallis Kendal and get a better picture of our process. We have good press, but very little money…lots of dedicated volunteers.
BY Gerry
ON September 8, 2010 06:11 AM
Yes, its great to have supportive local NGO’s helping you, but government partnerships are always a good idea, in my opinion. One resource (which of course depends on your religious affiliations) would be sunday school programs at larger churches, which, although not government funded, usually have many resources due to tax breaks. I wish you and New Futures all the best. I hope things have improved since 2007. -Gerry