Evaluation plan

From WikiResearcher
Jump to: navigation, search



Project clarification


Long term impact

The OER in Australia ALTC project is a working towards enhancing the efficacy of the Australian Higher Education system through the successful adoption and implementation of OER.

Focus of the project

The focus of this project is to research and develop a Feasibility Protocol in supporting higher education institutions in Australia in contributing to the impacts envisaged above.



Icon objectives.jpg
Aims
The primary aim of the project is to develop a "Feasibility Protocol" containing guiding principles to assist and support higher education institutions in Australia with their understanding of OER. Specifically, the Feasibility Protocol will contain a set of guiding principles with information on:
  • policy recommendations for higher education institutions in Australia regarding adoption, use and management of OERs;
  • what OERs are and the reasons why higher education institutions have adopted them;
  • the issues and barriers involved with the adopting of OERs, for example, copyright, intellectual property, licensing and other legal issues;
  • factors related to the use and management of OERs such as scope, purpose, strategic directions, institutional culture and resource allocation;
  • short case studies with examples from institutions that have adopted OERs (Who has adopted OERs and how have they used and managed them? What are the lessons learned?), and
  • the impacts on learning and teaching in higher education, as well as on informal and non-formal education.


Scope of the project

The project is divided into 2 phases to be conducted over a period of two years commencing in November 2010:

  • Phase 1: Investigation (November 2010 - October 2011): Data collection and preliminary analysis, including:
    • Literature study
    • Online survey
    • Focus group interviews.
    • Inventory of relevant policy documents from the international and national higher education sector.
    • Preliminary data analysis report
  • Phase 2: Development, evaluation and dissemination (November 2010 - October 2011): Drafting and refinement of the Feasibility Protocol, Consultation Symposium and dissemination of findings and project outputs, including:
    • Draft Feasibility Protocol
    • Face-to-face Consultation Symposium
    • Development of short case studies.
    • Revised and final version of the Feasibility Protocol
    • Report on project findings and "state-of-play" of OER in Australia.

The stakeholders specified in the project proposal include:

  • Tertiary educators and
  • Target audience for the final report including
  • Senior management (e.g. PVC(A)s, Directors of ICT, and Associate Deans of Teaching and Learning).

Evaluation purpose and scope

The purpose of this evaluation is to:

  • Develop a log frame, including output and outcome statements to inform the project team and reference group in monitoring and refining the project design.
  • Formative evaluation of the progress of the project in accordance with the evaluation criteria specified in the project proposal.
  • Summative assessment of the attainment of the project outcomes.

The evaluation plan is guided by the processes specified in the project submission and the extent that these have been achieved. There is a formative component to the evaluation plan where suggestions for improvement derived from the evaluation will be fed back into the project process. Where variations in the processes occur the reasons will documented and reported on including any unanticipated outcomes. Where practible, output metrics are monitored in accordance with the specified outcomes of the project. Value judgements on the qualitive aspects of the project will rely on the knowledge, skills and experience of the OER reference group convened for the project.

Project stakeholders and study audiences

The evaluation plan and reporting is designed for the following primary audiences:

  • Project team
  • Project reference group

As an open project, the evaluation plan and reports will be published under open content licenses and freely available for audiences interested in the progress of the project.

Logic model for the project

Generic log frame

Results Indicators Means of Verification Assumptions and risks

Impact

Increased adoption and implementation of OER at higher education institutions in Australia to:

  • reduce cost,
  • widen access and
  • improve quality of higher education.
Baseline data for current integration / utilistion of OER in Australia specfied.
  • Project Survey data
  • Analysis of institutional polices
Monitoring and evaluation plan does not incorporate impact analysis taking the constraints of the existing project into account. No post-project review is planned for this evaluation.
Outcomes
Project completed successfully
  • Project is completed on time within budget.
  • Project decisions are taken in consultation with the reference group.
Project processes and management
  • Review of statement of expenditure
  • Review of minutes of the reference group against the project plan.
  • Monitoring and assessement of the discussion lists and page metrics of the research wiki pages for the project.
  • Semi-structured telephone interview with selected members of the project team and reference group

Budget

Low risk for budget overspend in relation to the completion of the project as the host instution carries the risks of overspend for the contractual obligations of this grant contract.

Communication

Low risk. VIrtual communication and open project planning and execution in the wiki will ensure open and transparent communication.

Feasibility protocol published


Dissemination of findings


Outputs
Consulation symposium Consult output statements
  • Appreciative enquiry with project team and symposium facilitator
  • Semi-structured telephone interview 3 participants
  • Review of symposium documentation, rapporteer summaries and proceedings
  • Sample review of audio recordings of small group discussions.
  • Evaluation questionnaire to be completed at the symposium.
Medium risk: Project does not succeed in achieving adequate representation from the stakeholders given time committments of senior managers.
Draft feasibility protocol Consult output statements
  • Voting and approval of the feasibility protocol in the wiki
  • Monitoring of drafting and relevant discussions in the wiki and project discussion forum.
  • Semi-structured telephone interview with two leaders not directly involved in the study to evaluate utility of the feasibility protocol.
Low risk: Depth and breadth of featisbility protocol may exceed the scope of of the project resources.
Annotated bibliography Consult output statements
  • Mini-evaluation survey of the annotated bibliography distributed to all members of the reference group.
  • Review of minutes of the relevant project team and reference group meetings.
The expertise and experience of the OER reference group is deemed adequate to express a value judgement on the quality of the annotated bibliography.
Interview report Consult output statements
  • Sampled review of transcripts and/or audio/video recordings of the focus groups.
  • Review of planning documentation and report of the focus group interviews.
  • Appreciative enquiry with two participants from different focus groups.
Medium risk: procedural requirements of focus group interviews may exceed the resources and capacity of the project. Clear definition of the scope and purpose of focus groups needed.
Online survey report Consult output statements
  • Minutes and/or virtual approval by the Reference Group of the  survey instrument prior to implementation.
  • Minutes of the Reference Group meeting approving acceptance of the survey report.
Medium risk: Low response rates and challenges associated with gaining a representative sector wide opinion from responsdents who may not have an organisation-wide or systemic knowledge of the higher education system. Limitations of the methodoly must be assessed and clearly articulated in the report.

Activities (Process)




Literature study 80% of the project teams' specified objectives are achieved or exceeded.
  • Appreciative enquiry teleconference(s) with project team (formative evaluation focus)
  • Project team wiki-report on the achievement of specified objectives
Literature survey is a specified project outputs and more detailed summative review is planned. Low risk and appreciative enquiry teleconferences will identify any challenges early in the process.
Online survey
80% of the project teams' specified objectives are achieved or exceeded.
  • Appreciative enquiry teleconference(s) with project team (formative evaluation focus)
  • Project team wiki-report on the achievement of specified objectives
See corresponding comments in outputs section above.
Interviews 80% of the project teams'process related objectives are achieved or exceeded.

  • Appreciative enquiry teleconference(s) with project team (formative evaluation focus)
  • Project team wiki-report on the achievement of specified objectives
Focus group interview report is a specified project output and more detailed summative review is planned. Low risk and appreciative enquiry teleconferences will identify any challenges early in the process.
Inventory of relevant policies 80% of the project teams' process related objectives are achieved are achieved or exceeded.

  • Appreciative enquiry teleconference(s) with project team (formative evaluation focus)
  • Project team wiki-report on the achievement of specified objectives
Low risk and appreciative enquiry teleconferences will identify any challenges early in the process.
Data analysis 80% of the project teams' data analysis objectives are achieved or exceeded.
  • Appreciative enquiry teleconference(s) with project team (formative evaluation focus)
  • Project team wiki-report on the achievement of specified objectives
Low risk. Skilled project team and experienced reference group. Appreciative enquiry teleconferences will identify any issues early in the process.
Sharing and communicating findings


Inputs
  • Knowledge, skills and exprience of the project team and reference group
  • ALTC project grant
  • Published international research on OER
  • Existing institutional and other policy documentation


Criteria for judgements

Key evaluation questions and evaluation criteria (specified in project proposal)

Criterion Key questions Notes /Data requirements

Effectiveness of data collection processes in achieving project aims

  • Does the survey address the key areas identified in the pre-study?
    1. "Not-invented here" bias
    2. Institutional barriers
    3. Contextual barriers
    4. Discoverability
    5. Sustainability
  • Have survey participant recruitement processes been successful in engaging identified stakeholders  in the research project?
  • Have appropriate ethics approvals been gained and adhered to in the project?
  • Draft survey
  • Mapping and identification of key stakeholders
  • Draft ethics approval documentation
    • Check for clearances / ethics / anonymity on institutional policy documents
Identification of key themes and trends
  • How effective have the data collection methods been in identifying key trends, experiences and preferences in relation to adoption, use and management of OERs?
  • Initial themes and key trends may not be vailitated by the detailed literature survey. May need to recalibrate in the lights of the survey.
Appropriateness of framework guidelines in relation to the themes and trends identified in the data
  • What lessons have been learned from this project and how might these be of assistance to other institutions, researchers and decision-makers interested in the “feasibility framework” for adopting, using and managing OERs within their institutions?
  • To what extent have the"feasibility freamwork guidelines achieved their intended goal?
  • Goals of "feasibility framework"?

Effectiveness of dissemination strategies

  • How effective was the symposium in providing further feedback, evaluation and refinement of the framework guidelines?
  • What has been the impact of the dissemination strategy?
  • Note - Feasibility framework is an input for the symposium
  • Project must prepare dissemination strategy and targets
Effectiveness of project processes
  • What changes/amendments need to be made to ensure the project meets its intended aims?
  • Any variations from initial proposed processes, if so why?
  • What unintended benefits accrued from the project?
  • What factors helped/hindered the achievement of the outcomes?
  • Ongoing monitory through this evaluation plan will address this criterion.
Timeliness
  • Were timelines managed appropriately?
  • What strategies were in place for risk management?
  • Project must develop a risk management strategy?
On budget
  • Did the project come in on budget? Was the budget a fair representation of the resource requirements of the project?
  • Need access to budget documentation.

Dissemination of findings

  • Monitoring activities to be published openly on the project wiki pages
  • Reports to be tabled at reference group meetings
  • Interim report (October 2011) and final report (October 2012) to be published on the project wiki.

Evaluation timeline

Evaluation timeline corresponds and synchronises with the project plan.

Review of the evaluation plan

Ongoing review and refinement of the evaluation plan on the project wiki.