[governance-vn] A (International) Consultant to conduct project mid-term evaluation
Vern Weitzel
vern.weitzel at gmail.com
Sat Jun 14 08:48:43 EST 2008
A (International) Consultant to conduct project mid-term evaluation
Publish Date: 13-Jun-2008
The Project "Viet Nam: Promoting Energy Conservation in Small and Medium Scale
Enterprise" (PECSME project) is a joint effort of UNDP/GEF and the Government of
Viet Nam, aiming at encouraging application of the energy efficient practices
among SMEs in Vietnam, and through that contributing to the reduction of
greenhouse gases (GHG) emission among the sector. The project implementation
commenced in 2005 and is expected to complete in 2010.
The mid term evaluation of the project is required by the GEF/UNDP. The MTR aims
at determining the relevance, impact, effectiveness, efficiency and
sustainability of the interventions and contributions of the involved partners
and provide recommendations for necessary adjustment in project implementation
in the remaining period.
The United Nations Development Programme (UNDP) is looking for one (01) highly
qualified and experienced international candidate to conduct the project
mid-term evaluation assignment. Further information about the project can be
obtained by visiting http://www.undp.org.vn or sending an inquiry to the below
address.
Attached are the Terms of Reference for project mid term evaluation .
All interested international candidates are invited to send an application
letter and CV in English, indicating clearly “Application for International
Consultant – MTE PECSME” to the address below.
Special Note on Application: Candidates must be independent from both the
policy-making process and the delivery and management of the UNDP/GEF
assistance. Therefore, those who have any direct involvement with the design and
implementation of the PECSME Project will not be considered.
Contact
Ms. Nguyen Minh Ha
Human Resources Unit
UNDP Viet Nam
25-29 Phan Boi Chau Street
Ha Noi – Viet Nam
Fax: (84 4) 942 2267
Email: registry.vn at undp.org
Deadline for submissions: Friday 05:00 PM | 11-Jul-2008
Mid Term Evaluation - Promoting Energy Conservation in Small and Medium
Enterprises in Vietnam
Final Draft TOR MTE – PECSME
REV5 April 25 1
TERMS OF REFERENCE
PROJECT MID-TERM EVALUATION
Project title: PIMS 2057 CC FSP: Vietnam Promoting Energy Conservation in Small
and
Medium Enterprises
Project ID: 00045088
Implementing
Partner: Ministry of Science and Technology (MOST)
Duration of the
Assignment: 20 working days during 1 September - 30 October, 2008
Duty Location: Hanoi (Viet Nam) with in-country travel to project sites
1. Introduction
1.1. Country Context
Since early 2000, the government has paid attention to reducing the pressure on
energy supply by
issuing the Government Decree 102/2003 to set foundation for energy conservation
and energy
efficiency (EC&EE). Later, the government invested in the 2006-2013 National
Targeted
Programme on Energy Saving and Energy Efficiency, of which energy efficient in
industrial sector
is one important aspect.
UNDP aims to help the Government of Viet Nam to meet this challenge by helping
to build GOV
capacity to effectively conserve its scarce energy resources and make economic
growth more
efficient while fulfilling country’s obligations as a signatory to global
multilateral environmental
convention agreements (MEAs).
1.2. Project Summary
The PECSME project is a joint initiative of the Global Environment Facility
(GEF), the United
Nations Development Programme (UNDP), the Government of Vietnam and a number of
partners
involved in the selected priority industrial sectors. The goal of the project is
the reduction in the
annual growth rate of GHG emissions from the five selected SME sectors through
widespread and
sustained improvement in energy use. The project purpose is the removal of
identified barriers to
the widespread uptake of energy conservation technologies, improved management
focus on
reducing energy costs, enhanced technician training in energy conservation
practices and improved
operation of energy using equipment.
The project comprises an integrated set of activities designed to address in a
holistic fashion the
identified barriers to widespread utilization of energy efficient management
practices, operations
and technologies in Small and Medium Enterprise (SME) sectors in Vietnam.
SMEs are a key target for energy conservation in Vietnam as they now account for
95% of
enterprises, provide 26% of employment, contribute up to 25% GDP and are
expected to fuel
Vietnam’s future employment growth. The five key SME sectors being addressed in
the project are
brick, ceramics, textiles, paper and food processing. The project will take the
results of ten carefully
targeted demonstration projects in the five SME sectors and apply them to a
pipeline of 500
projects to be implemented during the project lifetime.
Mid Term Evaluation - Promoting Energy Conservation in Small and Medium
Enterprises in Vietnam
Final Draft TOR MTE – PECSME
REV5 April 25 2
It is estimated that by the end of PECSME project, that the successful
implementation of the project
will result in the cumulative energy saving of 136.1 KTOE and the cumulative CO2
annual
emission reduction of 962.0 ktonnes during the period 2006-2010.
To total project expected cost is USD 28.769 million with USD 5.469 million of
GEF financing and
USD 23.3 million from Vietnamese counterparts, including USD 19.1 million from
the business
sector (financial institutions, manufacturers and SMEs). Government ministries
and agencies fund
the difference. The GEF funding includes a provision of USD 1.950 million for
providing a loan
guarantee support to selected SMEs. The loan guarantee facility is explicitly
designed to be self-
sustaining beyond the end of the project.
1.3 Project Expected Outcomes and Outputs
The PECSME project is expected to reach its goals of GHG emission reduction and
EC targets by
producing six major outcomes as follows:
Outcome 1: Improvement of EC&EE Awareness and Building Capacity on EC&EE Policy
Development aiming to aiming to increase impact of existing policies and the
recently enacted
EC&EE decree through strengthened capacity of relevant government Ministries,
Departments
and Agencies in effective policy and institutional design, guidance,
implementation and
enforcement of energy conservation measures.
Outcome 2: Implementation of an adapted Communications Strategy with the aim
of Enhancing
SME and public awareness of EC&EE through an integrated communications system
including
information collection, dissemination and reporting.
Outcome 3: Development and Enhancement of EC/EC Technical Capacity with the
aim of
Improving technical and financial skills in EC&EE implementation through a
comprehensive
training plan.
Outcome 4: Development of EC/EE Technical support and services fostering of a
competitive,
high quality and sustainable energy efficiency services offer in Vietnam.
Outcome 5: Development of banking and financial sector’s awareness of benefits
of EE/EC
investments and willingness to finance SMEs through a better understanding of EE
projects and
by providing a loan guarantee, if needs be.
Outcome 6 : Demonstration of techno-economic feasibility of EC/EE projects
with the aim of
increasing credibility of EC&EE investments through successfully implemented and
evaluated
demonstration and replication projects.
The project should achieve these outcomes by supporting an integrated set of six
component
programs comprising: (1) Policy and institutional support development; (2)
Communications and
awareness; (3) Technical capacity development; (4) Energy efficiency services
provision support;
(5) Financing support; and (6) Demonstrations.
2. Project Status
While the conception of the project dates back to 2004, the project was formally
launched in
October 2005 (Prodoc signature) but started in the field by the Inception
Workshop in January
2006. The project expected duration is 5 years and should end in October 2010.
Mid Term Evaluation - Promoting Energy Conservation in Small and Medium
Enterprises in Vietnam
Final Draft TOR MTE – PECSME
REV5 April 25 3
All project components (related to Outcomes 1 to 6 above mentioned) are under
way even some
activities have been launched in late in regards of the initial planning:
training program
(Component 3) is under way of final completion up to 2009; technical support to
Energy Efficiency
Service Providers (EESPs) (Component 4) for improving EC/EE and projects
implementation
involves several EESPs working as sub-contractors; the financial support
(Component 5) is
operational for providing the requested loan guarantee, and all demonstration
projects (10) are
already completed (part of Component 6). Other soft components related to
Outcomes 1 and 2 are
ongoing.
The 2008 Annual Work Plan has been approved in early January 2008 and all
planned activities are
in progress. The challenge the project is currently facing is, among other
related to the
implementation of 500 replication projects (second part of Component 6).
3. Objectives of the Mid-Term Evaluation
The Monitoring and Evaluation Policy at the project level in UNDP/GEF has two
overarching
objectives:
a) Promote accountability for the achievement of GEF objectives through the
assessment of
results, effectiveness, processes and performance of the partners involved in
GEF activities.
GEF results will be monitored and evaluated for their contribution to global
environmental
benefits; and
b) Promote learning, feedback and knowledge sharing on results and lessons
learned among the
GEF and its partners, as basis for decision-making on policies, strategies, program
management, and projects and so to improve knowledge and performance.
The evaluation is to be undertaken in accordance with the “GEF Monitoring and
Evaluation Policy”
(see
http://thegef.org/MonitoringandEvaluation/MEPoliciesProcedures/mepoliciesprocedures.html).
The Mid-Term Evaluation is intended to assess the relevance, performance,
results, management
arrangements, and the overall achievement of the project. It looks at signs of
potential impact and
sustainability of results, including the contribution to capacity development
and the achievement of
global and national environmental goals.
The evaluation should provide evidence-based information that is credible,
reliable and useful,
enabling the timely incorporation of findings, recommendations and lessons into
the decision-
making processes.
The Mid Term Evaluation will be coordinated by the UNDP Viet Nam and supported
by the Project
Management Unit. The Mid Term evaluation will determine progress being made towards
objectives and will identify the required adjustments or improvements, if needs
be. The Evaluator
should include in his/her report all needed recommendations and lessons learnt
related to MTE
main findings with the aim of improving the project performance and of
delivering the final outputs
towards project objectives.
Mid Term Evaluation - Promoting Energy Conservation in Small and Medium
Enterprises in Vietnam
Final Draft TOR MTE – PECSME
REV5 April 25 4
4. Scope of the Mid Term Evaluation
The scope of the MTE covers the entire UNDP/GEF-funded PECSME project and its
components.
It aims to evaluate the project implementation taking into account the actual
status of the project
activities, related outputs and the resource disbursements made up to date. The
evaluation will
involve analysis at two levels:
- Components level, and
- Project level.
4.1 Component level
At the Component level, the evaluation will determine the relevance,
effectiveness, efficiency
and impacts of activities/components design, implementation, management, and
monitoring of
evaluation. Any issue or factor that has impeded or accelerated the
implementation of any of its
components, including actions taken and adjustments made should be analyzed and
highlighted.
On the component level, the following key aspects shall be evaluated:
4.1.1 The overall integration of the different Project components to make that
all synergies
and utilized
- How is the level of coherence and effective inter-linkage and communication
between and among project components in term of sharing information and
supporting each other towards achievement of the project objectives?
- Has there been any overlapping or synergy among activities which could make the
implementation more efficient?
4.1.2 Appropriateness of Key Project Indicators (KPI)
- Are outcome performance indicators and targets specific, measurable, achievable,
reasonable and time-bound?
- To what extent are the performance measurement indicators useful for dealing with
issue of quality management of the outputs and outcomes?
- To what extent shall the performance measurement indicators be improved or
adjusted for a better and more useful monitoring and evaluation continuous process?
4.1.3 Effectiveness of the Project Monitoring and Evaluation Process
- How effective is the project monitoring and evaluation process to ensure the
relevance and effectiveness of the activities and expected results in relation
to TORs
(RFPs) issued, different levels of work plans (QWP and AWP), and the required
outputs
quality criteria?
- Review the appropriateness and the actual use of tools the project laid out
for monitoring
and evaluating the progress of activities and to assess the project outputs and
outcomes in
term of technical quality, timeliness, and user’s expectations.
- To what extent is the M&E procedure used by project manager and component
coordinators to monitor and assess the actual progress of activities and
resulting outputs
and outcomes over the planned periods?
Mid Term Evaluation - Promoting Energy Conservation in Small and Medium
Enterprises in Vietnam
Final Draft TOR MTE – PECSME
REV5 April 25 5
4.1.4 Effectiveness utilization of deliverables from project
partners/stakeholders to
achieve the component outputs and project impacts
- Are the consultants/subcontractors/service providers engaged in project
implementation
suitably qualified, experienced and skilled to perform assigned responsibilities?
- Did the project management apply appropriate basic/standard rules/criteria to
select
currently engaged consultants/subcontractors?
- How does quality of the completed output meet the expectation of the project
(quality
criteria, scope of work, schedule)
4.1.5 Effective information sharing within the PMU and towards the UNDP country
office
- Assess the means and tools the PMU has implemented for establishing timely and
effective relationship and communication between project component coordinators
in such a way sub-activities are tightly synchronized, data, lessons learned and
best
practices are appropriately shared.
- To what extent is the UNDP CO adequately informed and is in a position for
getting access to updated information in regards to components implementation
schedule, deliverables from subcontractors, major outputs, impacts and delays or
other pitfalls the PMU encounters in the course of the components implementation?
4.1.6 Design and Project Components Appropriateness
- To what extent is the set of originally designed components in accordance with
the
project objectives and do the adjustments made from that time relevant to the
project
overall approach?
- Point out any changes the Project made in components design and/or expected
outputs/outcomes. Assess how do they affect the achievement of the whole project
outcomes and targets?
4.2 Project level
On the project level, the evaluation will focus among other, on four priority
topics1 in regards
to project performance:
(a) Progress towards objectives, achievement of results, targets set for the
entire project.
(b) Factors affecting successful implementation and achievement of results.
(c) Project Management framework, arrangement and performance,
(d) Strategic partnerships, and
(e) Project budget management.
4.2.1 Progress towards achievement of results
- Is the Project making satisfactory progress in achieving project outputs
vis-à-vis the
overall and immediate targets and in accordance with related inputs and activities
undertaken? Review of the mid-term actual results with reference to the approved
targets
1
Note: not all of the five criteria need to be systematically reviewed in all
aspects of project performance
Mid Term Evaluation - Promoting Energy Conservation in Small and Medium
Enterprises in Vietnam
Final Draft TOR MTE – PECSME
REV5 April 25 6
in terms of: (i) number of SMEs currently benefiting and other SMEs already
identified
into the EC/EE projects pipeline; (ii) energy savings and GHG emission reduction
(over
the project timeframe) resulting from the EC projects already implemented; (iii)
number
of provinces with the new EC technology in place.
- Given the level of achievement of outputs and related inputs and activities to
date, is
the Project likely to achieve its Immediate Purpose and Development Objectives and
its overall targets (EC, GHG and number of SMEs)?
- Are there critical issues relating to the achievement of project results that
have been
pending which would need immediate attention and/or action in the second half of
the
project implementation timeframe (up to 2010).
- In accordance with results got until now, are the project
partners/stakeholders able,
available and in a position to provide quality and in time inputs to achieve
results
within the remaining time until 2010?
4.2.2 Factors affecting successful implementation and achievement of results
Analysis should focus on factors beyond the Project’s immediate control or
project-design
factors that influence the project outcomes and results:
- Are there any outstanding issues, unexpected obstacles, bottlenecks, etc… from
the
energy end-users, government policy, business and financial sectors that could
affect
or jeopardize the successful implementation and achievement of project results on
time and in accordance with expectations? As an instance, to what extent the
current
economic context in Vietnam, especially the issue related to interest rate in
Vietnam,
could have an impact on SMEs’ EC project investment decision making process and
the use of the LGF?
- To what extent does the broader policy environment conducive to achieving
expected
project results, including existing and planned EE/EC legislations, decrees,
regulations, policy guidelines and government priorities?
- Is the project logical framework and design still relevant in the light of the
project
experience to date? Pinpoint any aspects of the ‘’logframe’’ that shall be
revisited and
updated and, if necessary, provide suggestion for timely changes or adjustments to
activities and time-bound targets.
- Do the Project’s purpose and objectives remain valid and relevant, or are
there items
or components in the project design that need to be reviewed and updated or simply
deleted?
- Are the current institutional and implementation arrangements still relevant and
helpful in the achievement of the project’s objectives, or are there any
institutional
concerns that are restrictive to project implementation and progress?
4.2.3 Project management (adaptive management framework)
- General implementation and project management: Assess the adequacy and
appropriateness of management arrangement in term of well performing the
management functions in order to successfully implement a full size project like
PECSME.
- Adequacy of management procedures and organization chart: Is the project
management arrangement appropriate to the extent of management functions, processes
and procedures, in accordance with the staff capacities and reasonable workload?
Is the
project organization chart efficient for conducting and managing each component,
and
the whole project on the technical and administrative perspective as well?
Mid Term Evaluation - Promoting Energy Conservation in Small and Medium
Enterprises in Vietnam
Final Draft TOR MTE – PECSME
REV5 April 25 7
- Evaluate the effectiveness of the project management and reporting system and
procedures in terms of: (i) timeliness of activities reporting and planning on
time, (ii)
relevance of the quality management plan of major inputs and outputs, (iii)
appropriateness of implementation of M&E plan and its internalization by the
project
component coordinators in conducting their activities. To what extent do the M&E
procedures provide sufficient basis for evaluating performance and decision
making at
the activity and outcome levels? How has the APR/PIR process helped in monitoring
and evaluating the project implementation and achievement of results?
- Likely Risks assessment and mitigation: does the project take into
consideration the
likely implementation risks in preparing the annual work plan and quarterly work
plans
with the aim of mitigating the impacts that could result from unexpected
situation or
change in the project environment? Review and discuss the risks originally
identified
in the project document and, the ones currently highlighted in preparing the AWP &
QWP. Assess the actual usefulness and timeliness of the original project logical
framework and work plans as management tools and in meeting with UNDP-GEF
requirements related to planning and reporting activities.
- Assess the use of electronic information and communication technologies in the
implementation and management of the project within the project team and through
project partners.
4.2.4 Strategic partnerships (project positioning and leveraging)
- Resource mobilization: has the current project management strategy exploited all
opportunities for strengthening collaboration and substantive partnerships with
other
government bodies, institutes, different associations, other donors, financial
sector
and other investment funds with the aim of maximizing project’s achievement of
immediate results and extending the project impacts in the long run beyond the end
of the project timeframe (2010)?
- Feedback from partners and relevant stakeholders: Are the project implementation
planning information, actual results status, and other priority concerns (e.g. :
like
implementation risks assessment) disseminated on time and on a regular intervals to
priority project partners? Are there areas to improve in the partnership mechanisms
for making more efficient the communication link and the timely information
dissemination and partners’ feedback related to priority topics and concerns?
4.2.5 Project budget management
- Disbursement calendar: Assess how activities were implemented against financial
disbursement planning. How were the budget provisions allocated, the timeliness of
disbursements and calendar, and the compliance of the procurement schedule
(services and equipment).
- On the project disbursement, report on the actual project costs (disbursement per
Component – Table below).
Mid Term Evaluation - Promoting Energy Conservation in Small and Medium
Enterprises in Vietnam
Final Draft TOR MTE – PECSME
REV5 April 25 8
Actual Budget Status in USD at the end of Q2 2008
Planned
Activities
Budget
Actual
Accomplishment
(%) NOTE 1
As per Project
Document
for the whole
project
timeframe
Actual
Disbursements Actual
vs. Project
Budget
Planning
(%)
Project Component 1
Project Component 2
Project Component 3
Project Component 4
Project Component 5
Project Component 6
Project Component 7
TOTAL
Finally, the Evaluation Report will present recommendations and lessons learned
that could
be helpful for improving the project performance and to reach its general
objectives and
particular targets in regards of GHG emission reduction, energy savings, and the
number of
EC replication projects.
5. EVALUATION METHODOLOGY
5.1. General
The MTE Consultant is expected to become familiar to the project objectives,
historical
developments, institutional and management mechanisms, activities and updated
status of
accomplishments. In order to make an evaluation of the project towards requirements
mentioned in the Section 4, the MTE is expected collect and analyze relevant
information
through:
- Review of original project documents and other documents related to PECSME and
EE&EC program and policy in Viet Nam;
- Group and individual interviews with project managers/officers and Senior
Technical
Advisors, component coordinators, relevant stakeholders, and at the least
representatives of the project partners and beneficiaries, and
- Project demonstration sites visits.
The analysis of the information should enable the Evaluator to make evidence-based
assessment of the priority topics mentioned in the Scope. For all topics
mentioned in
Section 4, the MTE Evaluator must provide the rating as described below. The
rating must be
supported by concrete evidence, e.g. narrative justification, data and statistics.
Definition of rating the project performance:
- Highly Satisfactory (HS): there were no shortcomings
- Satisfactory (S): there were minor shortcomings
- Moderately Satisfactory (MS): there are moderate shortcomings
- Moderately Unsatisfactory (MU): there were significant shortcomings
- Unsatisfactory (U): there were major shortcomings
- Highly Unsatisfactory (HU): there were severe shortcomings
Mid Term Evaluation - Promoting Energy Conservation in Small and Medium
Enterprises in Vietnam
Final Draft TOR MTE – PECSME
REV5 April 25 9
5.2 Evaluation process
An outline of an evaluation approach is provided below; however it should be
made clear that the
evaluator is responsible for revising the proposed approach as necessary. Any
changes should be in-
line with international criteria and professional norms and standards (as
adopted by the UN
Evaluation Group – Annex 3). They must be also cleared by UNDP before being
applied by the
Evaluator.
The evaluation must provide evidence-based information that is credible,
reliable and useful. It
must be easily understood by project partners and recommendations must be
applicable to the
remaining period of project duration.
In essence, the evaluation will be mainly supported by the following documents:
- PECSME Project documents and Inception Report (2005)
- Annual Work Plan (AWP 2006, 2007 and 2008)Quarterly Work Plan (Q1-Q2-Q3 2008),
QPR
(Q1-Q2 2008)
- Annual Project Progress Reports/Project Implementation Reports for Year 2006,
2007
- Budget planning and updated disbursements tables
- Guideline for Selection and Implementation of EC&EE project
- Loan Guarantee Agreement
- List of final technical outputs per component and sample of documents
(Walkthrough Audit
reports, Energy Audits reports, Feasibility Studies, Implementation reports, and
Verification
reports), if needs be.
- Demo Program evaluation report (2008)
- Financial Mechanism (LGF) Evaluation Report
- Other relevant documents the Evaluator may consider useful for evidence-based
assessment.
The evaluator is expected to follow a participatory and consultative approach
ensuring close
involvement from the government counterparts (MOST) and relevant project
partners. As an
instance, the Evaluator should be required to conduct interviews and meeting
with selected
stakeholders including, the NPD and UNDP PO, the Project Manager, PECSME
National and
International Senior Technical Advisor, project component coordinators(4),
director of selected
DOST and selected ESEPs, Incombank HQ LGF manager, selected SMEs’ manager
including site
visits and representative of Association of Small and Medium Enterprises in
Vietnam and other EC
fund managers like the Environment Protection Funds or others, if needs be.
The methodology used by the evaluator should be presented in detail in the
evaluation report. The
report shall also include in appendices information related to:
List of documentation reviewed;
List and timetable of interviews and site visits;
Questionnaires (if needs be);
Terms of Reference of MTE
Although the Evaluator should feel free to discuss with the authorities
concerned, all matters
relevant to its assignment, it is not authorized to make any commitment or
statement on behalf of
UNDP or GEF or the project management.
The Evaluator should reflect sound accounting procedures and be prudent in using
the resources of
the evaluation.
Mid Term Evaluation - Promoting Energy Conservation in Small and Medium
Enterprises in Vietnam
Final Draft TOR MTE – PECSME
REV5 April 25 10
6. DELIVERABLES
The output of the mission will be the Evaluation Report. Preferably the length
of the Report
should not exceed 30 pages in total (not including the annexes). The report
should be submitted
with quality that meets the evaluation quality standards provided with this TOR.
The Evaluator is required to prepare the followings documents:
- Mission agenda
- Questionnaires, if needs be
- Mission Main Findings Report, by the end of his (her) site presence;
- Draft Report after his (her) site presence
- Final Report
A two-week mission to Vietnam including selected project site visits will be
conducted.
7. TIMING AND DURATION
The total effort for conducting the evaluation has been estimated to 20 working
days within the
period starting from 1 September to 30 November 2008, according to the following
planning:
Preparation (home office – from 1 September to 12 September):
- Collection of and acquaintance with the project document and other relevant
materials with
information about the project;
- Lay out the detailed evaluation scope and mission agenda (including the
methods for data
collection and analysis); the agenda should be preferably sent to the UNDP and
PMU by
September 8 2008. The two-week site presence (working days in the field) should
be scheduled
from 15 to 26 September 2008.
- The Project Manager will arrange transportation for the consultant; will
arrange for
translation/interpretation when necessary, and if needs be, the PMU will book
and provide the
air flight ticket for conducting interviews in HCMC or visiting some selected
sites outside
Hanoi.
- Communication with the project staff to clarify matters.
Mission to Vietnam (10 working days in Vietnam from 15 to 26 September 2008):
- briefing with the UNDP PO
- briefing with the PMU
- project site(s) visits
- interviews and meetings with stakeholders
- review of additional project documents
- preparation of the mission main findings report, including the preliminary
rating of activities,
recommendations and lessons learnt
- debriefing with UNDP PO
- debriefing with the NPD and the PMU
- final mission main findings report and recommendations.
Elaboration of the draft report (home office till 13 October 2008):
- Additional desk review
- Completing of the draft report
- Presentation of draft report for comments and suggestions
Mid Term Evaluation - Promoting Energy Conservation in Small and Medium
Enterprises in Vietnam
Final Draft TOR MTE – PECSME
REV5 April 25 11
- Additional information and further clarification with UNDP, project management
and project
staff;
Elaboration of the final report (home office till October 30):
- Incorporation of comments and additional findings into the draft report
- Finalization of the report.
The draft Evaluation report shall be submitted to UNDP for review within 10
working days after
the mission. UNDP and the NPD will submit comments and suggestions within 10
working days
after receiving the draft report.
The final Evaluation Report shall be submitted latest on 30 October 2008.
8. REQUIRED QUALIFICATION
The evaluator must be independent from both the project design process and the
delivery of
assistance within the PECSME framework. The candidates will be evaluated in
accordance with the
criteria as follows:
- University degree in technical, economics or energy/environment related issues;
- Recent experience with result-based management evaluation methodologies;
- Recent experience in evaluation of international donor driven projects;
- Recognized expertise in the field of energy efficiency and related technologies;
- Familiarity with Energy efficiency in industrial sector;
- Familiarity with energy efficiency policies framework in a context similar to
Vietnam;
- Experience with financing mechanisms, financial analysis and evaluating
financial viability
of energy efficiency programs;
- Work experience in relevant areas for at least 10 years;
- Conceptual thinking and analytical skills;
- Project evaluation experiences within United Nations system or with the GEF
will be
considered an asset;
- Excellent English communication skills;
- Computer literacy;
If selected, failure to make disclosures as required in section 8 will be
considered just grounds for
immediate contract termination, without compensation. In such circumstances, all
notes, reports
and other documentation produced by the evaluator will be retained by UNDP.
Mid Term Evaluation - Promoting Energy Conservation in Small and Medium
Enterprises in Vietnam
Final Draft TOR MTE – PECSME
REV5 April 25 12
Annex 1
Evaluation Report: Sample Outline
(Designed for adaptation to specific project circumstances.)
Executive summary
Brief description of project
Context and purpose of the evaluation
Main conclusions, recommendations and lessons learned
Introduction
Purpose of the evaluation
Key issues addressed
Methodology of the evaluation
Structure of the evaluation
The project(s) and its development context
Project start and its duration
Problems that the project seek to address
Immediate and development objectives of the project
Main stakeholders
Results expected
Evaluation and Conclusions
Evaluation at the Component level
Evaluation at the Project level
Conclusion
Recommendations
Corrective actions for the design, implementation paln and calendar,
monitoring and evaluation of
the project
Actions to follow up or reinforce initial benefits from the project
Proposals for future directions underlining main objectives
Lessons learned
Best and worst practices in addressing issues relating to relevance,
performance and success
Annexes
Itinerary
List of persons interviewed
List of documents reviewed
Questionnaire used and summary of results
TOR
Mid Term Evaluation - Promoting Energy Conservation in Small and Medium
Enterprises in Vietnam
Final Draft TOR MTE – PECSME
REV5 April 25 13
Annex 2: UNDP Evaluation Report Quality Standards2
Standard 1: The title page and opening pages should provide key basic
information.
1.1 The following information should be easily accessible in the first few pages
of the report:
− name of the subject (i.e. activity, programme, policy etc.) being evaluated;
− date;
− table of contents, including annexes;
− name and organization(s) of the evaluators;
− name and address of the organization(s) that commissioned the evaluation.
Standard 2: The evaluation report should contain an Executive Summary.
2.1 An Executive Summary should provide a synopsis of the substantive elements
of the evaluation
report. To facilitate higher readership, the Executive Summary should be short,
two to three pages, and
should “stand alone”. The level of information should provide the uninitiated
reader with a clear
understanding of what was found and recommended and what has been learned from
the evaluation.
2.2 The Executive Summary should include:
− a brief description of the subject being evaluated;
− the context, present situation, and description of the subject vis-à-vis other
related matters;
− the purpose of the evaluation;
− the objectives of the evaluation;
− the intended audience of the report;
− a short description of methodology, including rationale for choice of
methodology, data
sources used, data collection and analysis methods used, and major limitations;
− the most important findings and conclusions;
− main recommendations.
Standard 3: The subject being evaluated should be clearly described, including
the logic model and/or
the expected results chain and intended impact, its implementation strategy and key
assumptions.
3.1 The evaluation report should clearly describe what the purpose of the
subject being evaluated is and
how the designers thought it would address the identified problem. Additional
important elements include:
the importance, scope and scale of the subject being evaluated; a description of
the recipients / intended
beneficiaries and stakeholders; and budget figures.
3.2 The description of the subject being evaluated should be as short as
possible while ensuring that all
pertinent information is provided. If additional details are deemed necessary,
a description including the
logic model can be provided in an annex.
Standard 4: The role and contributions of the UNDP, UN organizations and other
stakeholders to the
subject being evaluated should be clearly described.
2
These quality standards are drawn from the UN Evaluation Standards, UNEG, 2005.
Mid Term Evaluation - Promoting Energy Conservation in Small and Medium
Enterprises in Vietnam
Final Draft TOR MTE – PECSME
REV5 April 25 14
4.1 The report should describe who is involved, their roles and their
contributions to the subject being
evaluated, including financial resources, in-kind contributions, technical
assistance, participation, staff time,
training, leadership, advocacy, lobbying, and any contributions from primary
stakeholders, such as
communities. An attempt should be made to clarify what partners contributed to
which outcome.
4.2 Users will want to compare this with who was involved in the evaluation to
assess how different
points of view were included.
Standard 5: The purpose and context of the evaluation should be described.
5.1 The purpose should discuss why the evaluation is being done, how it will be
used and what
decisions will be taken after the evaluation is complete. The context should be
described in order to provide
an understanding of the setting in which the evaluation took place.
Standard 6: The evaluation report should provide an explanation of the
evaluation criteria that were
used by the evaluators.
6.1 Not all criteria are applicable to every evaluation. The rationale for not
using a particular criterion
should be explained in the report, as should any limitations in applying the
evaluation criteria. Performance
standards or benchmarks used in the evaluation should also be described.
6.2 It is important to make the basis of value judgments transparent.
Standard 7: The evaluation report should provide a clear explanation of the
evaluation objectives as
well as the scope of the evaluation.
7.1 The original objectives of the evaluation should be described, as well as
any changes made to the
evaluation design.
7.2 The scope of the evaluation should be described, making the coverage of the
evaluation explicit.
The limits of the evaluation should also be acknowledged.
7.3 The original evaluation questions should be explained, as well as those that
were added during the
evaluation. These are critical references against which the content of the
report ought to be compared to.
7.4 The objectives and scope of the evaluation are also critical references to
judge whether the
methodology selected and resources allocated were adequate.
Standard 8: The evaluation report should indicate the extent to which gender
issues and relevant
human rights considerations were incorporated where applicable.
8.1 The evaluation report should include a description of, inter alia:
− how gender issues were implemented as a cross-cutting theme in programming,
and if the
subject being evaluated gave sufficient attention to promote gender equality and
gender-
sensitivity;
− whether the subject being evaluated paid attention to effects on marginalized,
vulnerable and
hard-to-reach groups;
− whether the subject being evaluated was informed by human rights treaties and
instruments;
Mid Term Evaluation - Promoting Energy Conservation in Small and Medium
Enterprises in Vietnam
Final Draft TOR MTE – PECSME
REV5 April 25 15
− to what extent the subject being evaluated identified the relevant human
rights claims and
obligations;
− how gaps were identified in the capacity of rights-holders to claim their
rights, and of duty-
bearers to fulfil their obligations, including an analysis of gender and
marginalized and
vulnerable groups, and how the design and implementation of the subject being
evaluated
addressed these gaps;
− how the subject being evaluated monitored and viewed results within this
rights framework.
Standard 9: The applied evaluation methodology should be described in a
transparent way, including
any limitations to the methodology.
9.1 A comprehensive, but not excessive, description of the critical aspects of
methodology should be
contained in the evaluation report to allow the user(s) of the evaluation to
come to their own conclusions
about the quality of the data. Any description of the methodology should include:
− data sources;
− description of data collection methods and analysis (including level of
precision required for
quantitative methods, value scales or coding used for qualitative analysis);
− description of sampling (area and population to be represented, rationale for
selection,
mechanics of selection, numbers selected out of potential subjects, limitations
to sample);
− reference indicators and benchmarks, where relevant (previous indicators,
national statistics,
etc.);
− evaluation team, including the involvement of individual team members;
− the evaluation plan;
− key limitations.
The annexes should include the following:
− more detail on any of the above;
− data collection instruments (surveys, checklists, etc.);
− system for ensuring data quality through monitoring of data collection and
oversight;
− a more detailed discussion of limitations as needed.
Standard 10: The evaluation should give a complete description of stakeholders’
participation.
10.1 The level of participation of stakeholders in the evaluation should be
described, including the
rationale for selecting that particular level. While not all evaluations can be
participatory to the same
degree, it is important that consideration is given to participation of
stakeholders, as such participation is
increasingly recognized as a critical factor in the use of conclusions,
recommendations and lessons. A
human rights-based approach to programming adds emphasis to the participation of
primary stakeholders.
In many cases, this clearly points to the involvement of people and communities.
Also, including certain
groups of stakeholders may be necessary for a complete and fair assessment.
Standard 11: The evaluation report should include a discussion of the extent to
which the evaluation
design included ethical safeguards where appropriate.
11.1 The report should have a good description of ethical considerations,
including the rationale behind
the evaluation design and the mechanisms to protect participants where
appropriate. This includes
protection of the confidentiality, dignity, rights and welfare of human
subjects, including children, and
respect for the values of the beneficiary communities.
Mid Term Evaluation - Promoting Energy Conservation in Small and Medium
Enterprises in Vietnam
Final Draft TOR MTE – PECSME
REV5 April 25 16
Standard 12: In presenting the findings, inputs, outputs, and outcomes /
impacts should be measured to
the extent possible (or an appropriate rationale given as to why not).
12.1 Findings regarding inputs for the completion of activities or process
achievements should be
distinguished clearly from outputs, outcomes and impact.
12.2 Outcomes and impacts should include any unintended effects, whether
beneficial or harmful.
Additionally, any multiplier or downstream effects of the subject being
evaluated should be included. To the
extent possible, each of these should be measured either quantitatively or
qualitatively. In using such
measurements, benchmarks should be referred to.
12.3 The report should make a logical distinction in the findings, showing the
progression from
implementation to results with an appropriate measurement and analysis of the
results chain, or a rationale as
to why an analysis of results was not provided.
12.4 Data does not need to be presented in full; only data that supports a
finding needs to be given, and
full data can be put in an annex. Additionally, reports should not segregate
findings by data source.
12.5 Findings should cover all of the evaluation objectives and use the data
collected.
Standard 13: Analysis should include appropriate discussion of the relative
contributions of
stakeholders to results.
13.1 Results attributed to the subject being evaluated should be related back to
the contributions of
different stakeholders. There should be a sense of proportionality between the
relative contributions of each,
and the results observed. This is an integral element of accountability to
partners, donors and primary
stakeholders.
13.2 If such an analysis is not included in the report, the reason why it was
not done should be clearly
indicated. For instance, if an evaluation is done early in the management
cycle, results or any link to a
stakeholder’s contribution may not be found.
Standard 14: Reasons for accomplishments and difficulties of the subject being
evaluated, especially
constraining and enabling factors, should be identified to the extent possible.
14.1 An evaluation report should go beyond a mere description of implementation
and outcomes and
include an analysis, based on the findings, of the underlying causes,
constraints, strengths on which to build
on, and opportunities. External factors contributing to the accomplishments and
difficulties should be
identified and analysed to the extent possible, including the social, political
or environmental situation.
14.2 An explanation of context contributes to the utility and accuracy of the
evaluation. An
understanding of which external factors contributed to the success or failure of
a subject being evaluated
helps determine how such factors will affect the future of the subject being
evaluated, or whether it could be
replicated elsewhere.
Standard 15: Conclusions need to be substantiated by findings consistent with
data collected and
methodology, and represent insights into identification and/or solutions of
important
problems or issues.
Mid Term Evaluation - Promoting Energy Conservation in Small and Medium
Enterprises in Vietnam
Final Draft TOR MTE – PECSME
REV5 April 25 17
15.1 Conclusions should add value to the findings. The logic behind conclusions
and the correlation to
actual findings should be clear.
15.2 Conclusions must focus on issues of significance to the subject being
evaluated, determined by the
evaluation objectives and the key evaluation questions. Simple conclusions that
are already well known and
obvious are not useful, and should be avoided.
15.3 Conclusions regarding attribution of results, which are most often
tentative, require clear detailing of
what is known and what can plausibly be assumed in order to make the logic from
findings to conclusions
more transparent, and thereby increase the credibility of the conclusions.
Standard 16: Recommendations should be firmly based on evidence and analysis,
be relevant and
realistic, with priorities for action made clear.
16.1 For accuracy and credibility, recommendations should be the logical
implications of the findings and
conclusions. Recommendations should also be relevant to the subject being
evaluated, the Terms of
Reference and the objectives of the evaluation, and should be formulated in a
clear and concise manner.
Additionally, recommendations should be prioritized to the extent possible.
16.2 Recommendations should state responsibilities and the time frame for their
implementation, to the
extent possible.
Standard 17: Lessons, when presented, should be generalized beyond the
immediate subject
being evaluated to indicate what wider relevance they might have.
17.1 Not all evaluations generate lessons. Lessons should only be drawn if they
represent contributions
to general knowledge. They should be well supported by the findings and
conclusions of the evaluation.
They may refine or add to commonly accepted lessons, but should not be merely a
repetition of common
knowledge.
17.2 A good evaluation report has correctly identified lessons that stem
logically from the findings,
presents an analysis of how they can be applied to different contexts and/or
different sectors, and takes into
account evidential limitations such as generalizing from single point
observations.
Standard 18: Annexes should be complete and relevant.
18.1 Additional supplementary information to the evaluation that should be
included in annexes includes:
− list of persons interviewed (if confidentiality allows) and sites visited;
− data collection instruments (copies of questionnaires, surveys, etc.);
− the original Terms of Reference for the evaluation;
− list of abbreviations.
18.2 The annexes increase the usability and the credibility of the report.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 10795_080613_pecsme_mte.pdf
Type: application/pdf
Size: 117039 bytes
Desc: not available
Url : http://mailman.anu.edu.au/pipermail/governance-vn/attachments/20080613/8b18ecf8/attachment-0001.pdf
More information about the governance-vn
mailing list