How do we evaluate our training program content?

[KC9DDI]

There's a couple of discussions going on across the forum right now with a shared theme: A perception of a lack of an ability to have concerns, questions and suggestions regarding the Scouting program heard and addressed by folks at the National level.

 

To hopefully avoid another nonsensical thread lamenting how National is physically impossible of doing anything correctly, I'd like to focus my question on the specific area of our training course content and syllabuses (syllabi?) Pertaining to NYLT, for example, I know that National has run pilot programs at a handful of councils prior to rolling out nation-wide changes. Presumably feedback from these trial courses was incorporated into the final syllabus.

 

My questions are, how are those trial councils identified? Are there any existing processes to become involved in the evaluating and revision of our existing training materials? What exactly is the process that National uses to evaluate its own program?

 

Now, I don't realistically expect an organization the size of the BSA to individually respond to each and every complaint, question or suggestion raised by it's volunteers. I also understand that it's not possible to make everyone in our organization 100% happy, 100% of the time.

 

What I'd like to be able to do is develop an understanding of the existing processes, before I start offering criticism or suggestions for improvement. Does anyone whose worked at a National level have any experience in such things?

[Oak Tree]

I think this is a good question.

 

I doubt that there is a lot of evidence-based effort going on here, although I might be persuaded otherwise if I saw enough data. I definitely think that rolling things out in pilot councils is a good way to get feedback. But I think you'd want to run pilot programs for a few years and see what the results look like in order to do a fair evaluation. A few quick pilots can help iron out any obvious kinks, but if you really want to know if the changes are effective, you'd want some way to measure the results.

[Engineer61]

The OP said, "Presumably feedback from these trial courses was incorporated into the final syllabus."

 

There are two layers to this type of feedback ... feedback from the initial trials (Beta Testing) and CQI (Continuous Quality Improvement) once the program is rolled out.

 

The problem with Beta Testing is that it is a cost-driven exercise. The initial parameter of the exercise is the cost of executing the test and rolling the changes back into the program. Quality of results (or magnitude of improvement) is secondary in Beta Testing.

 

As such, Beta Testing is designed to find the "gross" errors in the program that can be fixed easily, with little cost or time impact. However, if the sample size is insufficient, then there is a high likelihood that large (but not necessarily fatal) errors will be uncovered and fixed.

 

This is where CQI comes in. Feedback is received from every user of the product and the results are correlated and scored based upon severity and cost to remedy. This is probably where BSA falls short. Because we're talking about printed material and face-to-face training from the top down. Changes resulting from CQI are most likely prohibitively expensive to implement for all but the most glaring of changes. Changes that would be implemented would be those issues that are contrary to law, or that involve a major policy shift at National.

 

So the point here is that after the initial testing, there are likely to be few if any changes to parts of the program that are not initiated by a fundamental shift in policy by National. Of course, as you trickle down the food chain, there may be locally initiated changes to satisfy the needs of a given council/district/troop.