- 1 Project clarification
- 2 Evaluation purpose and scope
- 3 Project stakeholders and study audiences
- 4 Logic model for the project
- 5 Criteria for judgements
- 6 Dissemination of findings
- 7 Evaluation timeline
- 8 Review of the evaluation plan
Long term impact
The OER in Australia ALTC project is a working towards enhancing the efficacy of the Australian Higher Education system through the successful adoption and implementation of OER.
Focus of the project
The focus of this project is to research and develop a Feasibility Protocol in supporting higher education institutions in Australia in contributing to the impacts envisaged above.
Scope of the project
The project is divided into 2 phases to be conducted over a period of two years commencing in November 2010:
- Phase 1: Investigation (November 2010 - October 2011): Data collection and preliminary analysis, including:
- Literature study
- Online survey
- Focus group interviews.
- Inventory of relevant policy documents from the international and national higher education sector.
- Preliminary data analysis report
- Phase 2: Development, evaluation and dissemination (November 2010 - October 2011): Drafting and refinement of the Feasibility Protocol, Consultation Symposium and dissemination of findings and project outputs, including:
- Draft Feasibility Protocol
- Face-to-face Consultation Symposium
- Development of short case studies.
- Revised and final version of the Feasibility Protocol
- Report on project findings and "state-of-play" of OER in Australia.
The stakeholders specified in the project proposal include:
- Tertiary educators and
- Target audience for the final report including
- Senior management (e.g. PVC(A)s, Directors of ICT, and Associate Deans of Teaching and Learning).
Evaluation purpose and scope
The purpose of this evaluation is to:
- Develop a log frame, including output and outcome statements to inform the project team and reference group in monitoring and refining the project design.
- Formative evaluation of the progress of the project in accordance with the evaluation criteria specified in the project proposal.
- Summative assessment of the attainment of the project outcomes.
The evaluation plan is guided by the processes specified in the project submission and the extent that these have been achieved. There is a formative component to the evaluation plan where suggestions for improvement derived from the evaluation will be fed back into the project process. Where variations in the processes occur the reasons will documented and reported on including any unanticipated outcomes. Where practible, output metrics are monitored in accordance with the specified outcomes of the project. Value judgements on the qualitive aspects of the project will rely on the knowledge, skills and experience of the OER reference group convened for the project.
Project stakeholders and study audiences
The evaluation plan and reporting is designed for the following primary audiences:
- Project team
- Project reference group
As an open project, the evaluation plan and reports will be published under open content licenses and freely available for audiences interested in the progress of the project.
Logic model for the project
Generic log frame
|Results||Indicators||Means of Verification||Assumptions and risks|
Increased adoption and implementation of OER at higher education institutions in Australia to:
| Baseline data for current integration / utilistion of OER in Australia specfied.
|| Monitoring and evaluation plan does not incorporate impact analysis taking the constraints of the existing project into account. No post-project review is planned for this evaluation.|
| Project completed successfully
|| Project processes and management
Low risk for budget overspend in relation to the completion of the project as the host instution carries the risks of overspend for the contractual obligations of this grant contract.
Low risk. VIrtual communication and open project planning and execution in the wiki will ensure open and transparent communication.
|Feasibility protocol published||
|Dissemination of findings||
|Consulation symposium||Consult output statements||
|| Medium risk: Project does not succeed in achieving adequate representation from the stakeholders given time committments of senior managers.|
|Draft feasibility protocol|| Consult output statements
|| Low risk: Depth and breadth of featisbility protocol may exceed the scope of of the project resources.|
|Annotated bibliography|| Consult output statements
|| The expertise and experience of the OER reference group is deemed adequate to express a value judgement on the quality of the annotated bibliography. |
|Interview report|| Consult output statements
|| Medium risk: procedural requirements of focus group interviews may exceed the resources and capacity of the project. Clear definition of the scope and purpose of focus groups needed. |
|Online survey report||Consult output statements||
|| Medium risk: Low response rates and challenges associated with gaining a representative sector wide opinion from responsdents who may not have an organisation-wide or systemic knowledge of the higher education system. Limitations of the methodoly must be assessed and clearly articulated in the report.|
|Literature study|| 80% of the project teams' specified objectives are achieved or exceeded.
|| Literature survey is a specified project outputs and more detailed summative review is planned. Low risk and appreciative enquiry teleconferences will identify any challenges early in the process. |
| Online survey
|| 80% of the project teams' specified objectives are achieved or exceeded.
|| See corresponding comments in outputs section above.|
|Interviews|| 80% of the project teams'process related objectives are achieved or exceeded.
|| Focus group interview report is a specified project output and more detailed summative review is planned. Low risk and appreciative enquiry teleconferences will identify any challenges early in the process. |
|Inventory of relevant policies|| 80% of the project teams' process related objectives are achieved are achieved or exceeded.
|| Low risk and appreciative enquiry teleconferences will identify any challenges early in the process. |
|Data analysis|| 80% of the project teams' data analysis objectives are achieved or exceeded.
|| Low risk. Skilled project team and experienced reference group. Appreciative enquiry teleconferences will identify any issues early in the process.|
|Sharing and communicating findings||
Criteria for judgements
Key evaluation questions and evaluation criteria (specified in project proposal)
|Criterion||Key questions||Notes /Data requirements|
Effectiveness of data collection processes in achieving project aims
|Identification of key themes and trends||
|Appropriateness of framework guidelines in relation to the themes and trends identified in the data||
Effectiveness of dissemination strategies
|Effectiveness of project processes||
Dissemination of findings
- Monitoring activities to be published openly on the project wiki pages
- Reports to be tabled at reference group meetings
- Interim report (October 2011) and final report (October 2012) to be published on the project wiki.
Evaluation timeline corresponds and synchronises with the project plan.
Review of the evaluation plan
Ongoing review and refinement of the evaluation plan on the project wiki.