Data Collection

​Data will be collected from all the stakeholders involved in the program. The stakeholders in this program would include the:
  • Staff,
  • Students, and
  • Parents
In addition to collecting data from those who are directly involved in the program. I feel it is important to collect data from the evaluators perspective on the themes covered in the program. As a result of this the data would take on a more holistic approach and also question the evaluators assumptions and biases on the topics covered. According to Shulha & Cousins, knowledge is socially constructed and linked to context. (p. 203) In this approach this would encourage 'systemic inquiry' through collaborative modes.  As a result of this evaluators are effectively introduced in the program communities habits of systemic inquiry and reflective activity  associated to technical, analytical, and reporting skills. (Shulha & Cousins, 204)

how will Data be Collected?

Where it is possible a balance of both quantitative and qualitative data should be collected. By doing this analysis could be made subjectively and objectively. For each stakeholder the following approaches could be considered:

Staff:

Qualitative:
  • Focus Groups
  • Outcome Hierarchy through workshop style delivery

Quantitative:
  • ​Pre and post - questionnaires to program implementation​
Students:
​This project was conducted with Grade 10, 11 and 12 students of approximately 150 students.
Qualitative:
  • Project Diary - A random sample will be taken from 10 students in each grade level.
  • Focus Groups

Quantitative:
  • ​Pre and post - questionnaires to program implementation

Parents:

Qualitative:
  • Focus Groups
  • Problem Tree/Solution Tree Analysis
Quantitative:
  • ​Pre and post - questionnaires to program implementation

Evaluators:

Qualitative:
  • Focus Group within the evaluation team
  • Stakeholder Analysis

Quantitative:
  • ​Pre and post - questionnaires to program implementation and knowledge of community perspective
  • Time Tracking to oversee efficiency ​
Picture
Evaluation Toolbox https://www.evaluationtoolbox.net.au/index.php?option=com_content&view=article&id=62&Itemid=158#OHmap
Picture
Evaluation Toolbox: https://www.evaluationtoolbox.net.au/index.php?option=com_content&view=article&id=62&Itemid=158#OHmap
Picture
Example Gnatt Chat for time tracking - https://www.e-education.psu.edu/geog871/l5_p5.html

aNALYSIS & Evaluation Usability

Source:
Pritchard, M. (n.d.). Tool Selector. Retrieved from https://www.evaluationtoolbox.net.au/index.php?option=com_content&view=article&id=51&Itemid=131 
​
Shulha, L., & Cousins, B. (1997). Evaluation use: Theory, research and practice since 1986. Evaluation Practice, 18, 195-208.

Saunders, M. (2012). The use and usability of evaluation outputs: A social practical approach. Evaluation, 18(4), 421-436.
​Similar to the varying forms of student inquiry, analysis for this program design could also be designed to follow a similar pathway. As the data would need to be systematically collected, a more controlled and guided approach to the inquiry should be used. As indicated in the image to the left, I feel in an evaluation design there needs to be a balance between having a structured and an unstructured approach.

As a result of this, it would support organisational learning to be purposeful. In addition to this, with the aid of both qualitative and quantitative data, it allows the information which is generated for 'usability practices' as indicated by Saunders (2012). With this data analysis could be based on the following: (Saunders, 431)
  1. Reasons and purposes for the evaluation: With the use of the logic model of theory action and change a clear discussion can be conducted through the focus groups.
  2. The potential use of the evaluation: Through the use of a report and presentations, the data could be used to provide strategies or recommendations to inform strategic planning, staff/student/parent development and hiring of qualified staff members.
  3. The foci of the evaluation: Through the focus groups all stakeholders can co-create practices that could be implements to improve the program further.
  4. ​The nature of the data and evidence: The data set is both qualitative and quantitative would could better represent the community's perspective.
  5. Audience for the evaluation output: All stakeholders would be presented with the evaluation output and an opportunity to engage in dialogue for next steps.
  6. Timing of the release of the evaluation output: Timing of releasing evaluation output would take place at the beginning, middle and end to oversee the outcomes and impact of the program through a full cycle of delivery.
  7. Agency for undertaking the evaluation: The evaluation team would consist of both internal and external evaluators to have both 'old' and 'fresh' eyes look at the efficacy of the program.