Impact Assessment of ICT for Development Projects
Way back in 1992, when the world of ICT was still very nascent, the International Development Research Centre (IDRC), had initiated a research study to answer a fundamental question of the link between information and development, particularly to build upon the various experiential perspectives from case studies of projects implemented, and develop a common methodology for the diverse and wide-spread projects that they support.

Most supporting or donor agency perspectives for assessment of ICT for Development projects pertain to problems of sustainability, measuring the extent of empowerment to communities, linkage between ICT goals and development goals and accountability of institutions. Questions pertain to whether it can be demonstrated that the use of ICTs for Development is an essential resource for social and economic development of developing countries The potential value of the use of ICTs is still not very self-evident.

Expressions like “ICT is a critical resource and plays a fundamental role in development”, are axiomatic expressions. However neither data nor empirical evidence to support the above statement are possible unless one reviews the intricate issues in greater detail. Though several simple quantitative performance evaluation tools and methodologies exist to numerically measure a few indicators, many of the techniques do not go further than the statistics. Impact or ‘outcomes’ as they are often referred to, differ from ‘outputs’. This research has since evolved into a refined methodology called “Outcome Mapping” which is a process that combines monitoring, evaluation, learning and dynamic modifications to achieve the best impact.

What is Monitoring and Evaluation?
Monitoring is a systematic collection of data to provide management, donors and other stakeholders with an indication of project progress.

Evaluation is a systematic and objective assessment of on ongoing or completed project, including its design, implementation, and results. Evaluation leads to more informed decisions, allowing those involved in the project to learn from experience and to be accountable to donors and stakeholders.

Both Monitoring and Evaluation should be used to demonstrate accountability and to understand the dynamics of the program. The differences in the objectives, methodology and purposes of Monitoring and Evaluation are shown in the table below. It is important to keep in mind that Monitoring does not replace Evaluation.

There can be five types of assessment indicators:

1. Performance indicators, relating inputs to outputs
2. Effectiveness indicators, relating outputs to usage
3. Cost-effectiveness indicates relating inputs to usage
4. Cost-benefit indicators, relating inputs to outcomes
5. Impact indicators, relating usage to outcomes and domain characteristics.

After the assessment data is gathered, analysed, and the relationships between inputs, outputs, benefits/outcomes has been determined, a strategy must be developed for communicating the findings to the target audience(s). This is a critical part of an effective monitoring and evaluation strategy. Both quantitative and qualitative measures are possible. In ICT4D projects, often case studies and feedback reporting by beneficiaries/end-users of the project supplement quantitative analysis by the project implementers.


Source: Adapted from UNDP Handbook on Monitoring and Evaluating for Results
Evaluation can be done in four ways:

• Internal or self-evaluation, which means the same people implementing a project are responsible for evaluation
• External evaluation, meaning that the evaluation is conducted by an individual or group outside the implementing organisation
• Internal evaluation with the use of an external consultant, which is undertaken by the team implementing the project with the assistance of a professional evaluator. Such assistance can either apply to a specific component of a evaluation or to the entire evaluation process
• Independent evaluation, which is undertaken by individuals or groups who are not only outside the implementing organization, but also completely independent from it, in terms of control, remuneration capacity, political pressure, or other factors that could affect objectivity

Evaluations must be conducted using participatory techniques to be effective. In complex development projects with multiple stakeholder participation, and both technology and development objectives, the process must involve all stakeholders in feedback evaluations.

Depending on the focus and objectives of the project the perspective and mix of methodologies will vary. There are two interesting manuals that will assist project implementers to conceptualise and design an M&E strategy and plan. If an external consultant has to be hired, then the terms of reference must be clearly defined.

e-Governance Assessment Framework
The need for a rational assessment framework for e-Governance projects is important due to the following reasons:

• Significant investments of resources into e-Governance projects
• Need to move away from subjective assessments and value judgments
• The (Indian) National Action Plan on e-Governance (NeGP) has an ambitious plan ahead, and several new initiatives will be kicked off in the coming years, with potential for replication and up-scaling of successful experiments. There is therefore a critical need to assess all the implemented projects
• Chanelizing ongoing efforts in the right direction as several projects at different stages of implementation need a common framework to best utilize available resources
• Facilitating funding agencies to take a rational view, especially in view of huge private investments flowing in the government sector.

Quality of e-Governance project and services must be assessed
Based on this, an e-Government Assessment Framework has been developed to assess various e-Government projects.

The MDGs arose to counter-blast to the perceived failure of neo-liberal agenda, one that favours markets, the private sector, and globalisation – to deliver for the world’s poor. –Richard Heeks

The attributes to be assessed have been classified into:

• service orientation
• technology
• sustainability
• cost effectiveness
• replicability

The methodology so developed has been designed to serve many objectives spelt out in the document. It also recommends that independent external agents, who are specifically hired for this purpose, and who are experienced to do the task, carry out the assessments. The criteria and examples of agencies who can be engaged are also identified in the report. The categories of projects that have been outlined fall in four categories:

• Government to Citizen Urban Environment (G2C-U)
• Government to Citizen in Rural Environment (G2C-R)
• Government to Business (G2B)
• Government to Government (G2G)

Two levels of assessments, one quick, and another, in-depth are outlined to serve the impor-tant purpose of identifying which of the e-Governance projects should be replicated in other locations.

Two important attributes for this are:

1. Value that the project delivers to its primary clients and also to the many other stakeholders that are involved in delivering the government service. Though it is difficult to monetize the value, it still needs to be measured in concrete terms.
2. Adaptability of the Technology architecture to different contexts.

The Context of Millennium Development Goals
The Millennium Development Goals, established in 2000, was reviewed in September 2005. With 18 specific targets focusing on how ICTs can be useful in catalyzing the development processes to achieve global and national goals, and numerical indicators being agreed upon to measure the achievements of these targets, the stage is set to review whether ICTs have after all played a catalytic role or not! In an incisive analysis on the connection between ICT and Development, Richard Heeks, of the Development Informatics Group, University of Manchester, UK remarks, “The MDGs arose to counter-blast to the perceived failure of neo-liberal agenda, one that favours markets, the private sector, and globalisation – to deliver for the world’s poor.” He further notes that it is technology that generates wealth of enterprise, which in turn, pays for all social development.



A workshop is being organised by UNDP-Asia Pacific Information Development Programme, Thailand, during the World Summit on the Information Society at Tunis in mid November 2005. This is a partnership effort on measuring ICT4d initiated by the International Telecommunications Union (ITU), the OECD, UNCTAD, UNESCO, UN ICT Task Force, UN Regional Commissions, the World Bank and Eurostat. The objective of this collective partnership is to better understand how to build and use indicators for measuring Information and communications technologies for development (ICT4D). This event would bring together ICT stakeholders at national, regional and international levels to present a set of core ICT indicators recommended for national statistical offices. The panelists will discuss the importance of measuring the information society for ICT policy-making, and development in areas such as education, government and health.

Such a partnership is very critical to ensure that the evaluation and progress made due to support and coordinated efforts of various agencies and the very sparse international aid that has been committed to this sector can be measured in an effective way, and the impact of the ICTs on various development sectors can be measured. One of the ways would be an assessment of progress made based on the objectives. Another is to evaluation the impact or outcomes based on broad development goals. In the past five years, the Millennium Development Goals (MDGs) have been NISG and i4d reserve the right to reprint articles produced for the ICTD section of the i4d magazine and website, with due credits to NISG and i4d. Please write to the editor for any request of reprints.

References:

1. Interview with Salil Shetty, Millennium campaign gains momentum, i4d, p13-14, February 2005
2. ITU, 2004 The Impact of ICTs on the MDGs
3. Richard Heeks, ICTs and MDGs: on the wrong track? I4d, p9-11, February 2005
4. Measuring the impact of information on development, edited by Paul McConnell, IDRC, 1995
5. Sarah Earl, Fred Carden, Terry Smutylo Editors, Outcome mapping : building learning and reflection into development programmes Ottawa, International Development Research Centre (IDRC), 2002, 139 pages
6. Monitoring and Evaluation, Guidelines for Country Gateway Implementation Grants Funded by Development Gateway Foundation
7. E-Governance Assessment Frameworks (EAF Version 2.0) prepared by TP Rama Rao, V Venkata Rao, SC Bhatnagar, and J Satyanarayana, May 2004 http://egov.mit.gov.in/NISG_EAF_18-05-04.pdf