HRH-PDT Evaluation Strategy

Now please open another browser window to

   http://camss.clemson.edu/HRH_Pilot_Test/pdt.html                                                                         

and click on Policy Development.

This shows a table with 43 rows (the number of questions) and
8 columns (the number of policy outcomes).  Each cell has
two numbers which represent the contribution to the policy
outcome of the client's response to the question. The numbers
are weights that we are to assign to the client's response.
The upper number is the contribution to the outcome if the
client answers "Yes" (i.e., "Strongly Agree" or "Agree").
The lower number is the contribution to the outcome if the
client answers "No".  

I assigned the numbers you see in the table.  Take, for 
example, Question #5 ("There are many vacancies in the 
country for: Nurses/Midwives/etc."). If the client answers
"Yes" for Nurses, OnQ adds "1" to NursesSumA, the sum of
this client's responses for Nurses outcome "A" ("Increase
production capacity for Nurses"). If the client answers
"No", OnQ adds "-1" to (i.e., subtracts 1 from) NursesSumA. 

If the client answers 
"Don't Know", OnQ adds "0" to the NursesSumA.  In all 
three cases, OnQ will increment N, the total responses
of this client contributing to NursesSumA.  At the end
of the evaluation, OnQ will divide NurseSumA by N,
giving a ratio.  This ratio will be a number between "1"
(the client consistently gave answers that OnQ judged 
to suggest increasing the production capacity for Nurses)
and "-1" (the client consistently gave answers that OnQ 
judged to suggest *not* increasing the production capacity
for Nurses).  If you recall your basic statistics, this
analogous to the measure for correlation coefficient.

This ratio is the number that OnQ will report for NursesRatioA.
There will also be measures for MidwivesRatioA, DoctorsRatioA,
etc.  There will also be measures for outcomes B, C, ..., H.
The measures will be ranked from highest to lowest and will
be presented to the client, together with a summary of
the client's responses that led to the measures.  If the
client revisits the survey and changes some of his/her
answers, OnQ may present a different ranking.

Note that the weights of next question (Question #6 =
"Despite many vacancies in the job market for health workers, 
there are a large number of unemployed") are the reverse
of those for Question #5.  This is because, in my opinion,
if a large number are still unemployed despite vacancies,
the problem is *not* production, but something else 
entirely.  So a "Yes" in Question #6 effectively cancels
a "Yes" in Question #5.

The "Don't Know" answer presents a slightly different
view of the recommendations.  If, for example, ten questions
contribute to NursesRatioA. Furthermore, say the client
answers "Yes" (+1) to each of two of the questions, "No"
(-1) to one of the questions, and "Don't Know" to the 
other seven.  Then OnQ will report a ratio of 

           (1+1-1)/10 = 1/10 = 0.1  

However, OnQ will also report a confidence of 3/10 or 0.3, 
because only three of the ten questions were answered by 
the client.  In other words, OnQ's confidence in its 
recommendation is directly proprational to the amount of 
information that the client provides.  More information 
means greater confidence.  If the client responded with 
"Don't Know" to all questions, OnQ would report back a 
ratio of "0" but with a confidence of "0", the lowest 
possible.

Thus the client will know exactly how the measures and
the rankings were determined, and this will hopefully 
lead to a better understanding of how the shortages 
in his/her country are coming about and what he/she
can do about them.

One last point, the right column of "I. Policy Outcomes"
in the Structure link can now be explained.  The question
numbers of questions that contribute to the ratios for 
the different outcomes are listed beside each outcome.
In the report that will be presented to the client, these
question numbers together with the answers the client 
provided will be listed so that the client can see 
explicitly how the recommendations were developed.

============================================================

Conclusion:

   Please let us know what you think of this strategy for
calculating the recommendations.  There are other issues
we can  discuss (Should we use weights other than 1 and -1?
Should we allow fractional weights?
Should we give greater weight to "Strongly Agree" than
to "Agree"? Are there enough questions, especially for
Category 3?  Should questions be modified?).  However, 
please let us know whether or not you agree with the 
basic approach we are taking. If not, let's discuss it.

   But if you think that the strategy is sound, we are 
ready to implement this design.  What we would then need 
most from you and your staff are the response weights to put 
into the table.  The numbers you now see are all my *guesses*
as to what these weights would be for a few questions.  You
are much closer to the problem and can give a more considered
opinion of what these values should be.

Thanks,

Roy
 ------------------------------------------------ 
| Roy P. Pargas, PhD                 204 McAdams |
| pargas@clemson.edu            Computer Science |
| work: 864.656.5855          Clemson University |
| cell: 864.650.4771      Clemson, SC 29634-0974 |
| fax : 864.656.0145  www.cs.clemson.edu/~pargas |
 ------------------------------------------------