Atlantic Canada Opportunities Agency
Symbol of the Government of Canada
 

 

Evaluation of the Atlantic Canada Opportunities Agency’s 
Atlantic Policy Research Initiative

Evaluation Unit

Finance and Corporate Services

Atlantic Canada Opportunities Agency

March 17, 2010

 

ACOA
Atlantic Canada Opportunities Agency
APEC
Atlantic Provinces Economic Council
APRI
Atlantic Policy Research Initiative
AIF 
Atlantic Innovation Fund
CED-Q
Canada Economic Development for Quebec Regions
CURA
Community University Research Alliance
BDP
Business Development Program
ED
Enterprise Development
Gs&Cs
Grants and contributions
OECD
Organisation for Economic Co-operation and Development
O&M
Operations and maintenance
PAA
Program Activity Architecture
PAC
Policy, Advocacy and Coordination
PRI
Policy Research Initiative
RDA
Regional development agencies
R&D
Research and Development 
RPP
Reports on Plans and Priorities
SME
Small and medium-sized enterprise
S&T
Science and technology
SSHRC
Social Sciences and Humanities Research Council
Ts&Cs
Terms and conditions
VP
Vice-president

Executive Summary
  • 1.0 Introduction
  • 2.0 Evaluation Methodology  
  • 2.1 Study Limitations
  • 3.0 APRI Profile   
  • 3.1 ACOA Program Activity Architecture   
  • 3.2 APRI Objectives, Activities and Expected Outcomes
  • 4.0 Findings   
  • 4.1 Relevance      
  • 4.1.1  Key Findings – Relevance   
  • 4.2 Performance       
  • 4.2.1 Incrementality      
  • 4.2.2  Achievement of Expected Outcomes      
  • 4.2.3 Other Intended or Unintended Outcomes      
  • 4.2.4  Key Findings – Performance   
  • 4.3 Demonstration of Efficiency and Economy       
  • 4.3.1 Incrementality      
  • 4.3.2 Complementarity/Duplication/Overlap      
  • 4.3.3 Leveraging of Support from Funding Partners      
  • 4.3.4 Alternatives for Design and Delivery or Funding Mechanism      
  • 4.3.5 The APRI Recognized as an Effective Model      
  • 4.3.6 Key Findings – Efficiency and Economy
  • 5.0 Conclusions and Recommendations  
  • 5.1 Conclusions  
  • 5.2 Recommendations
  • 6.0 Alignment of Key Findings, Conclusions and Recommendations
Introduction

 List of Figures

  • Figure 1: ACOA’S 2007-2008 TO 2008-2009 PROGRAM ACTIVITY ARCHITECTURE
  • Figure 2: APRI LOGIC MODELFIGURE
  • Figure 3: ABILITY TO PROCEED WITH LESS FUNDING
  • Figure 4: ALIGNMENT OF CONCLUSIONS, KEY FINDINGS TO RECOMMENDATIONS

List of Tables

  • TABLE 1: APRI CLIENT-DRIVEN G&C APPROVED FUNDING
  • TABLE 2: APRI O&M PROJECTS VALUES
  • TABLE 3: APRI G&C AND O&M EXPENDITURES BY FISCAL YEAR
  • TABLE 4: APRI PROFILE OF RESEARCH ACTIVITIES BY TYPE
  • TABLE 5: SUMMARY OF KEY INFORMANTS 

 

Executive Summary

Introduction

This evaluation report presents the findings, conclusions and recommendations of the Atlantic Canada Opportunities Agency’s (ACOA) Atlantic Policy Research Initiative (APRI). The evaluation was designed to explore the core issues of relevance, performance and cost-effectiveness, and focused on the results achieved over the five-year period 2004-2005 to 2008-2009. 

 
The evaluation was conducted using multiple lines of evidence, which included the following six methodologies: (1) review of key documents/ literature, (2) review of 55 grants and contributions (Gs&Cs) and 35 operations and maintenance (O&M) paper files, (3) 16 in-depth interviews with ACOA staff and external experts in the area of policy research and mobilization, (4) database analysis using ACOA’s QAccess project database and GX financial database, (5) Internet survey with funding recipients, and (6) eight case studies.

The key limitations of the evaluation were related to its scope. The evaluation was limited to activities within APRI, and did not permit an overview analysis of APRI within the Policy, Advocacy and Coordination (PAC) program activity. Outcomes that extended beyond the five-year period covered by the evaluation could not be assessed. The achievement of outcomes can be influenced by factors outside APRI, creating a challenge with attributing successes or failures to the initiative. To the extent possible, strategies were implemented to address these limitations, and any conclusions made were supported by valid and reliable data.

APRI

Established in 2000, APRI is the Agency’s primary policy research and engagement funding program. It supports ACOA’s policy, advocacy and coordination functions and the Agency’s mandate, by working with external experts to conduct two types of projects: research and engagements. The projects need to be pan-Atlantic in nature, and contribute to the economic growth in Atlantic Canada. 

Summary of Key Findings

Relevance/Alignment with Government Priorities

  • The evaluation found that there is a demonstrable need for research activities conducted under APRI at    ACOA.The initiative plays a significant role in meeting policy research and engagement responsibilities identified by the Government of Canada – to conduct research in order to remain current with the evolving economy/environment,and to engage with external researchers to produce knowledge and to develop networks that will facilitate discussion of the findings.
  • APRI research and engagement projects are focused on the priorities of the Government of Canada   relating to economic development.They support the ACOA Program Activity Architecture, especially in terms of policy decisions and direction.

Performance

  • APRI was highly incremental (97% of projects would have sustained a major negative impact without ACOA funds). The provision of lesser funding would have been more detrimental to research than to engagement activities. If ACOA had provided less funding, the majority of research projects would not have proceeded, and engagements would have sustained negative impacts in areas such as quality, scope and completion.
  • The research and engagements funded under APRI have contributed to a better understanding of the regional economy, identified areas for support, resulted in specific recommendations, and provided a greater understanding of some strategic sectors. ACOA has been cited by external experts in policy research as being a leader among the regional development agencies (RDAs) in producing high quality research for evidence-based decision making.
  • The types of research supported by APRI include economic policy research (35.3% of research funds); sector studies (20.4%); innovation research (15.3%); community economic development research (11.1%); entrepreneurship and skills development research (9.4%); and trade and investment research (8.6%). APRI research and engagements have contributed to strategic plan formulation and program renewal, particularly in the area of innovation.
  • APRI has contributed to ACOA’s policy and programs direction, including innovation (e.g. Atlantic Innovation Fund), trade (e.g. Atlantic Gateway), enterprise development (e.g. Small and medium enterprise (SME) financing), and community development (e.g. impact of higher education institutions). However, ACOA programs staff and regional offices are not very familiar with APRI. While the initiative needs to remain responsive to external economic issues that influence the Agency’s mandate, it also needs to be informed by potential pan-Atlantic policy research topics arising from ACOA programs and regions. APRI management and staff could work more closely with head office programs staff as well as ACOA regional staff.
  • Projects funded under APRI have contributed to policy papers or advice to the minister. Knowledge generated through APRI is utilized to support advice to the minister either proactively (communication of results at the end of a project) or in response to questions originating from the minister’s office.
  • Knowledge produced through APRI is used by ACOA to advocate, particularly during engagement activities. The information resulting from these activities is used by stakeholders to more effectively address Atlantic Canadian issues in regional economic development. External stakeholders include provincial departments, federal departments, industry associations, other non-government organizations, the private sector and local governments.
  • All lines of evidence from this evaluation indicate that APRI has had a pan-Atlantic impact by enhancing policy research capacity in the region and by producing knowledge that is utilized by ACOA and other stakeholders such as provincial governments. APRI has also built and maintained a more focused, better coordinated network of research partners and stakeholders with common interests.

Efficiency and Economy

  • APRI does not duplicate other programs; minimal overlap has been identified with other research activities. The initiative complements other research conducted by provincial governments, universities, and think tanks such as the Atlantic Provinces Economic Council.  It also complements national research produced by organizations such as the Conference Board of Canada, Statistics Canada and the Policy Research Initiative, all of which have established collaborations with APRI.
  • Recipients of ACOA G&C funding through APRI have been successful at obtaining additional sources of financing for research and engagements. Analysis shows that for every dollar of APRI investment, an additional $1.21 is leveraged from other organizations, representing an increase of approximately 16% since the previous APRI evaluation, conducted in 2005. These results indicate that APRI is cost-effective in terms of leveraging impact. 
  • The evaluation identified opportunities for improving the effectiveness of program design and delivery of APRI. These included increasing promotion of APRI to new potential research and engagement collaborators; increasing communication of the initiative within the Agency; and establishing more inclusive processes for determining priority areas of research and project selection. Knowledge dissemination plans and activities for individual projects need to be recorded, as well as project outcomes for both G&C and O&M projects. 
  • Best practices identified for future projects include the involvement of key stakeholders in planning for engagements and research, and agreement on project deliverables with other funding partners from the outset.
  • Although some overlap exists between research projects conducted under the Business Development Program (BDP) and APRI, the BDP’s terms and conditions are not conducive to APRI activities.
  • APRI has been identified as a cost-effective policy research development model, and inspired the Canada Economic Development for Quebec Regions (CED-Q) to develop a business case for a similar policy research G&C funding program.

Conclusion – Relevance /Alignment with Government Priorities

APRI is relevant and meeting a demonstrated need for policy research in Atlantic Canada.  Engagement and research activities supported by the program are aligned with the Government of Canada’s priorities and ACOA’s areas of interest, as well as the Agency’s policy, advocacy and coordination functions.

Conclusion – Performance

ACOA plays a key role in producing economic policy research and related engagements in the Atlantic region. The Agency has been successful in meeting APRI intended outcomes. The initiative provides ACOA with the knowledge required to support policy development, advocacy and coordination efforts, and has built a reputation for excellence in policy research. One area identified for improvement would have the APRI management and staff work more closely with ACOA programs and regions to identify research needs.

Conclusion – Efficiency and Economy

APRI is considered to be cost-effective and to provide value for money. The evaluation results demonstrate that APRI activities are effective and emphasize the need for further development which could be achieved by increasing promotion to new researchers; enabling more inclusive processes for setting research priorities and selecting projects; and strengthening communications practices and performance measurement of the initiative.

Recommendations

This evaluation has identified opportunities for improvement, leading to the following recommendations to further the achievement of desired APRI outcomes.
 

  1. Implement more inclusive processes in setting priorities and selecting projects.
  2. Develop internal and external communications plans in order to promote APRI and disseminate knowledge.
  3. Keep an organized account of data on O&M projects, as these are subject to accountability requirements, along with the G&C projects to strengthen performance measurement. The data recorded should systematically include the rationale behind the dissemination strategy and parties to which the report was disseminated, irrespective of the medium (mail, e-mail, web links). Data on project outcomes should also be recorded.
  4. Foster the application of the following best practices identified in APRI delivery:
  1. the involvement of key regional stakeholders in planning for engagement and 
    research to increase regional relevance; and     
  2. the definition of project deliverables with funding partners from the outset.

1.0 Introduction

In accordance with requirements set forth in the Financial Administration Act and the Federal Accountability Act, as well as the Treasury Board’s Transfer Payments Policy and the Policy on Evaluation, the Atlantic Canada Opportunities Agency (ACOA) has conducted an evaluation of the Atlantic Policy Research Initiative (APRI). Given that the funding program represents minimal risk and expenditures, the evaluation approach and methodology were adapted accordingly. This report presents the findings, conclusions and recommendations of the APRI evaluation.

This evaluation examined issues of relevance and performance (effectiveness/efficiency/ economy), as stipulated in the Treasury Board’s Policy on Evaluation. The scope of the evaluation focused on the results achieved for APRI projects that were approved over the five fiscal years 2004-2005 to 2008-2009.

2.0 Evaluation Methodology

The APRI evaluation is based on a multiple lines of evidence approach to compensate for any methodology shortcomings and improve the reliability and validity of results. The methodology included a mix of qualitative and quantitative methods. Findings from each line of enquiry have been compared using a triangulation approach to support the findings. Conclusions are drawn based on, and contextualized by, the available evidence respecting sound evaluation principles. The methodologies are described as follows.

  • Review of key documents/literature.
  • Administrative data analysis including:
    •  QAccess project data on all 55 grants and contributions (G&C) projects approved
      between 2004-2005 and 2008-2009; and
    •  data from all 35 operations and maintenance (O&M) contracts extracted from the 
      ACOA GX financial database and housed within the APRI unit.
  • Review of 55 G&C and 35 O&M paper files obtained from ACOA central records.
  • Sixteen in-depth interviews with:
    • ACOA staff within and outside the policy area in head office and 
      ;regional offices; and
    • external experts from university policy research organizations 
      and a national policy research program.
  • Internet survey of 29 APRI funding recipients.
  • Eight case studies.

Additional details regarding the methodology can be found in Appendix A
 

2.1 Study Limitations

The evaluation design and implementation are considered appropriate based on the risk and materiality of APRI, the objectives of the study, and the application of multiple lines of evidence, which incorporated a mix of qualitative and quantitative methods. Nevertheless, some limitations are important to note. Where feasible, measures were taken to minimize these limitations, and strengthen the reliability and validity of the data. The key limitations encountered during the study are described below.

Evaluation Scope

The scope of this evaluation included only the activities conducted under the APRI funding program; therefore, it was not possible to assess APRI’s fit and appropriateness within the Policy, Advocacy and Coordination (PAC) program activity to which the APRI is aligned. A future evaluation of PAC will enable such evaluation questions to be answered.

File Content

A file review was completed for G&C and O&M projects; however, unlike G&C files, O&M files did not consistently contain supporting documents and final reports, particularly for engagement projects. This resulted in an inability to use the O&M file review to assess some evaluation questions. Some of these O&M projects were evaluated in more detail as part of case studies, where information additional to the file content was assessed.

Non-Response

Non-response bias occurs when the answers of non-respondents may differ from the answers of respondents. This evaluation included an Internet survey of APRI clients and contract providers. In order to understand non-responses, survey respondents were contacted and it was determined that a number of non-responses resulted from cases where an employee moved to another organization, which should not bias the results.

Long-Term Nature of Expected Impacts

The timeframe within which some outcomes of research and engagement projects occur can extend over several years. As such, case studies were carefully selected to provide insight on the manner in which APRI activities have led to longer term outcomes. They were also chosen to highlight best practices and lessons learned. The case studies are not statistically representative of the population of APRI projects. For projects that have been completed in recent years, some outcomes may not yet have occurred; this was taken into consideration during the analysis. For example, two projects were not yet fully completed (i.e. the engagement had not yet occurred or the research report was not yet produced). These were removed from the assessment of outcomes using file review, but included in the file review for all other evaluation questions.

Attribution of Results

Many of the APRI funding recipients had received financial support for multiple ACOA projects. In such cases, the APRI project that was the object of the survey was specified.
Many recipients of G&C funding had also received funds for a given project from other partners. In order to assess the impact of and need for ACOA funding, recipients were asked about their projects’ feasibility had ACOA funding had not been available (incrementality). The issue of outcome attribution also arises in the case where activities internal and external to APRI contribute to the same outcomes. External influences that can impede the success of APRI were identified by key informants and are part of the findings of this report. Rather than seeking to explore net impacts for this type of research and engagement funding program, this evaluation adopted a more appropriate approach used by other regional development agencies in the evaluation of similar programming – to solicit stakeholder views on whether APRI activities contributed to the achievement of desired outcomes. This methodological challenge was raised in the Evaluation of Policy, Advocacy and Coordination at Western Economic Diversification Canada (July 2009) [1] , and stakeholder views were utilized as a more appropriate approach.

3.0 APRI Profile

Established in 2000, APRI is the Agency’s primary policy research and engagement funding program. The purpose of APRI is to enhance the capacity of the Agency to coordinate and plan pan-Atlantic federal activities contributing to economic growth in Atlantic Canada[2]. Policy research and engagement activities supported through APRI can be described as follows.

Policy research activities are strategically focused on socio-economic issues affecting Atlantic Canada, and actively involve public and private sector partners and stakeholders from across the region and outside it. Research activities include targeted studies on key priority areas such as innovation and commercialization, productivity and competitiveness, trade and investment, skills development, and specific industry sectors. Policy research informs policy analysis, policy development, and advice on policy options and strategies. It also provides leading-edge insights and information for the Agency’s advocacy efforts, and is a source of strategic information for the Agency’s coordination mandate.

Engagement activities provide leaders from government, private sector and academia opportunities to participate in dialogue around key socio-economic issues affecting Atlantic Canada and the nation. Activities include policy roundtables and strategically focused conferences that bring new insights through discussion papers, new policy research and expert presentations on subjects aligned with current regional, Agency and government priorities. Engagements provide valuable input from regional and national leaders that informs Agency policy advice and general knowledge. These activities also contribute directly to the Agency’s advocacy efforts and can be an important vehicle in support of coordination efforts [3] .
 
The unique feature of APRI within ACOA’s policy function is that it supports externally produced research and engagements, thereby harnessing research expertise in the region other than what exists within ACOA, and enabling stakeholders such as industry associations to engage with other key players to exchange knowledge on policy topics and further support the Agency’s mandate for economic development in Atlantic Canada.  As such, these activities complement the Agency’s other policy research and analysis activities that are conducted internally by ACOA staff. 

3.1 ACOA Program Activity Architecture

The ACOA 2007-2008 to 2008-2009 Program Activity Architecture (PAA) included PAC as one of three strategic outcomes that supported the Agency’s mandate – to enhance the growth of earned income and employment opportunities in Atlantic Canada (see Figure 1).

The PAC strategic outcome was described in the PAA as resulting in policies and programs that strengthen the Atlantic economy. The PAA positioned APRI in support of Policy, described as follows: “The policy function is carried out by ACOA officials at its head office, regional offices, and the Ottawa office. It is supported by internal and external research on significant Atlantic economic issues: through ongoing analysis of issues and trends, challenges, and opportunities facing the region; and through stakeholder engagement. ACOA’s policy function is supported by a dedicated research program, the APRI, which funds externally produced region-wide research projects, and is designed to contribute to building the critical mass of public policy research capacity in Atlantic Canada [4] .”

                  Figure 1: ACOA’s 2007-2008 to 2008-2009 Program Activity Architecture

 2007 2008 2008 2009 PAA

 Source: ACOA PAA 2007-2008 and 2008-2009

ACOA’s PAA was revised effective April 1, 2009, and PAC was established as a program activity supporting the Agency strategic outcome of “A competitive Atlantic Canadian economy”. This change did not impact the positioning of APRI in supporting externally produced policy research and engagement activities within the policy function, that in turn support PAC.

3.2 APRI Objectives, Activities and Expected Outcomes

The APRI objectives are:

  • To enhance ACOA’s capacity to develop, manage and deliver federal economic development activities  in Atlantic Canada, including the development of  strategic approaches that  take into account regional economic opportunities and challenges;
  • To support and influence the federal priorities, including horizontal issues,that impact ;the Atlantic region, particularly those directly related to economic development; and
  • To establish and promote networks with other stakeholders.

 The logic model in Figure 2 identifies the APRI activities, outputs, reach and expected outcomes (immediate, intermediate and ultimate). The main objectives, as stated above, are reflected in the expected outcomes of APRI; therefore, this evaluation was limited to the assessment of the immediate and intermediate outcomes versus the objectives, to avoid duplication of information. The scope of the evaluation did not permit the assessment of the ultimate outcome.

                                     Figure 2: APRI Logic Model

 Figure 2 APRI Logic Model

 Source: Results-based Management and Accountability Framework (2005)

3.3 APRI Funding Sources

APRI utilizes two streams of funding for its research and engagement activities: (1) client-driven projects funded through G&C funds and (2) ACOA contracts obtained through O&M funds, explained in further detail below.

3.3.1 Grant and Contribution Projects

For client-driven projects supported by G&C funds, proponents submit an application to conduct a research project or engagement activity. Proposal are evaluated against the APRI project selection criteria and assessed by management. Once a project is selected, the head office policy analysts negotiate contributions with the client to be the least amount required to allow a project to proceed; this must conform to the Treasury Board Secretariat’s Policy on Transfer Payments. The level of federal assistance to projects under APRI should not exceed 90%. The remaining 10% can be qualified as in-kind contribution from either the recipient or other partners.

Projects should have an Atlantic Canada scope (i.e. impact more than one Atlantic province); contribute to the APRI objectives, ACOA’s program activities, and other priorities such as business climate, competitiveness and growth, emerging sectors, human capital and social economy; contribute to ACOA’s capability to carry out its planning and coordination responsibility (e.g. conducting new research, promoting greater coordination between federal and provincial economic development policies, contributing to the reduction of duplication and overlap in the processes of implementing federal and provincial economic development policies); and be conducted in conjunction with at least one or more public or private sector partners or participants for outreach activities such as roundtables, workshops and conferences.

The maximum federal amount payable to any one project under APRI is $300,000, and the maximum amount payable to any one recipient over a five-year period is $1 million. Designated Agency officials approve projects under delegated authority of the G&C funding program.

Approval of client-driven G&C research and engagement projects is outlined in Table 1. Between 2004-2005 and 2008-2009, ACOA approved funding for 55 G&C APRI projects, consisting of 30 research projects and 25 engagements, for a total of over $2.89 million.
 

                          Table 1: APRI Client-Driven G&C Approved Funding

 

Fiscal Year of Approval Number of G&C Projects ACOA Approved ($) 

Research

 Engagement

Total

2004-2005

9

3

13

702,730

2005-2006

6

7

13

374,850

2006-2007

8

2

10

456,620

2007-2008

4

9

13

595,862

2008-2009

3

4

7

345,115

Total

30

25

55

2,898,777

Total Approved ($)

1,715,790 1,182,987

2,898,777

 

Average Approved ($)

57,193

47,319

52,705

                           Source: QAccess, November 2009

3.3.2 Operations and Maintenance Projects

In some circumstances, ACOA identifies a need for a project that requires outside expertise for which no client has submitted a proposal. As such, APRI O&M funds are used to develop contracts for desired services through ACOA’s procurement function.  APRI O&M funds may also be utilized to enter into collaborative arrangements or memorandums of understanding (MOUs) with other partners, including other federal departments, generating a horizontal initiative. In these cases, the activity requires a pooling of resources between federal departments to ensure the provision of the goods or services required to achieve a common objective. ACOA’s use of financial resources is most efficiently done through O&M funding. APRI management also utilizes the APRI O&M expenditures to purchase data or reports for the Agency.

The O&M funding mechanism is subject to the Treasury Board Secretariat Contracting Policy and ACOA’s Contracting for Services Policy and Procedures. Authorities are subject to the department’s and minister’s legal or legislative authorities.

APRI expenditures in O&M contracts for research and engagements are outlined in Table 2. Between 2004-2005 and 2008-2009, APRI contributed to 35 O&M projects, consisting of 18 research projects, 15 engagements and two institutional memberships, for a total contribution of over $453,237.

                              

                                      Table 2: APRI O&M Projects Values

 

 

Fiscal Year of Approval

# of O&M Contracts

APRI Contract $

Translation Admin $

Total O&M $

Research

Engagements

Membership

2004/05

5

1

0

119,799

52,084[5]

171,883
2005/06

5

3

0

110,647

31,222

141,869
2006/07

4

2

0

80,833

8,486

89,319
2007/08

2

3

1

84,854

0

84,854
2008/09

2

6

1

57,105

4,067

61,171
Total #

18

15

2

35

n/a

n/a

Total Value $

278,024

124,089

8,987

453,237

95,859

549,097

Average Value $

15,446

8,723

4,493

12,950

n/a

n/a

                    Source: APRI data records (November 2009)

A total of $95,859 in O&M expenditures were dedicated to translation of documents and other administrative costs, such as printing and the purchase of report covers. Reported costs for 2004-2005 mistakenly included a research contract of over $45,000, reducing the actual total of translation and administration costs over the five-year period to $50,678.

Total estimated expenditures for APRI are detailed in Table 3.

                     Table 3: APRI G&C and O&M Expenditures by Fiscal Year

 

 

Type of Funding

     Fiscal Year

Total $
2004/05 2005/06 2006/07 2007/08 2008/09

Other Operating $

171,883 141,869 89,319 84,854 61,171 549,097

Contrib $

360,068 721,822 622,897 640,661 383,742 2,729,190

Total $

541,585 847,729 662,217 700,139 400,438 3,278,287

                          Source: ACOA Corporate Finance (February 2010).

4.0 Findings

4.1 Relevance

Demonstrable Need/Alignment with Federal Roles and Responsibilities

As ACOA’s primary mechanism for externally produced policy research and engagements that support a pan-Atlantic focus, the Agency positioned APRI within its PAA under Policy, Advocacy and Coordination – one of the three program activities contributing to the Agency’s strategic outcome “A competitive Atlantic Canadian economy.” The APRI supports the Agency’s mandate, “To increase opportunity for economic development in Atlantic Canada to enhance the growth of earned incomes and employment opportunities in that region,” and the object of the ACOA Act, “… to support and promote opportunity for economic development of Atlantic Canada, with particular emphasis on small and medium-sized enterprises, through policy, program and project development and implementation and through advocacy of the interests of Atlantic Canada in national economic policy, program and project development and implementation.”

Over the period of 2004-2005 to 2008-2009, APRI has supported the funding of 48 research activities, for a total of over $1.98 million, and of 40 engagement-related activities, for a total of over $1.3 million (includes G&C and O&M funding).

A review of key documents reveals that the activities undertaken within APRI are needed and align with federal roles and responsibilities. The Policy Research Initiative (PRI) paper Capacity, Collaboration and Culture: The Future of the Policy Research Function in the Government of Canada elaborates on the status of policy research within the Government of Canada as well as the needs going forward.  The document states:

Policy research helps the public service to understand and address current and emerging policy issues by providing impartial evidence-based research that can inform the policy development process. The strength of federal policy research depends on ongoing leadership, access to current and relevant data, senior executive demand, strong analytical capacity, and stable funding…. Other forces are having an impact on how policy research in government is organized and conducted. These include the increasingly complex, interrelated and horizontal nature of policy issues requiring research support; global influences on policy formulation; demand for policy tailored to unique regional requirements; the need to produce policy research quickly to enable policy makers to respond effectively to a rapidly changing economy/environment, and accelerating demand for forward-looking insight[6] .”

Not only does the document highlight the need for policy research in Canada, it identifies a need for region-specific policy research. It also encourages government agencies and departments to collaborate on projects and develop networks that can discuss findings. These are key activities within APRI.

Similarly, a presentation by the Clerk of the Privy Council, entitled “Policy Making in the 21st Century: New Challenges for Canada,” outlined the critical role of social science and policy research in meeting new policy development challenges. The clerk stated that governments will need to work with partners inside and outside government to develop effective policy solutions. [7]

According to external experts interviewed, there is a critical need for policy research that produces evidence to drive the agenda. Of equal importance are activities that bring knowledge to the table and share it with key partners. APRI achieves this by sharing reports as well as engaging with key partners through engagement activities such as roundtables and conferences. External experts also state that ACOA must supplement its internal policy research capacity by supporting external researchers and think tanks in order to gain diverse and independent perspectives. This is thought to be increasingly important, as stated in the PRI paper cited above, and adds to the legitimacy of APRI activities. According to ACOA’s policy management and staff at head office and in the regions, APRI allows ACOA and project proponents to investigate issues that influence the continuously evolving economy of Atlantic Canada through research and roundtables. APRI permits ACOA to gather, consult and share information with key stakeholders around a particular policy issue. It was mentioned that up-to-date information on the economic context is necessary for ACOA to discover new areas of intervention and to ensure that current interventions remain relevant.

APRI activities were compared with the types of activities conducted by other RDAs and were found to be consistent (i.e. all policy, advocacy and coordination units of other RDAs are conducting research production and engagement activities).

Alignment with Federal Government Priorities, ACOA Interests and the PAC Strategic Outcome

The section above established the need for the federal government to conduct policy research and engagement activities.  Files were reviewed to assess if APRI research and engagement projects were utilized to produce knowledge relating to priorities identified by the federal government and that relate to the Agency’s mandate and areas of interest. All G&C projects reviewed were found to be related to either Government of Canada priorities or ACOA areas of interest.

In regard to alignment with Government of Canada priorities, 82% of APRI engagements explored issues related to federal priorities, with 73% of them exploring themes directly related to items identified in the document Advantage Canada: Building a Strong Economy for Canadians, including knowledge advantage (work knowledge, skills, R&D innovation, and environment), infrastructure advantage (building the modern infrastructure required for success), entrepreneurial advantage (reducing regulations, red tape and increasing competition in the Canadian marketplace), fiscal advantage (government debt) and tax advantage (tax reduction for new business investment). Other topics relating to Government of Canada priorities and strategies were the subject of research or engagements through APRI due to their importance for ACOA’s mandate of increasing opportunity for economic development in Atlantic Canada.  These topics include immigration (Canada’s Immigration Strategy), Public Service Renewal, Aboriginal populations (e-government for Aboriginal populations) and sustainable development. 

All files were aligned with ACOA areas of interest, with 93% of research and engagements exploring policy issues directly related to themes identified in ACOA’s Report on Plans and Priorities (RPP) for the years 2004-2005 to 2008-2009. That is to say, APRI projects explored issues relating to most of the themes identified in the RPPs, including international trade, labour market and skills development, productivity and competitiveness, natural resources, rural/urban issues, innovation, trade corridors, transportation, immigration, strategic sectors and Aboriginal development. Also, knowledge produced by APRI projects were utilized to develop five policy papers written by the Agency in 2007 to support future direction. Since then, related research has been conducted to further inform the issues identified. Of all files reviewed, 78% were on topics relating to the five policy papers on Productivity and Competitiveness, Labour Market Skills, Science and Technology, Natural Resources and the Knowledge Economy, and Urban-Rural issues.

As seen in this section, APRI supports engagements and events on a broad spectrum of topics that affect the Atlantic economy. These topics reflect priorities identified by the Government of Canada and by ACOA.  The knowledge is generated to guide the Agency in setting policy and program direction. As such, APRI produces knowledge that contributes to the PAC program activity expected results.

4.1.1  Key Findings – Relevance

Relevance/Alignment with Government Priorities
  • The evaluation found that there is a demonstrable need for research activities conducted under the APRI at ACOA. The initiative plays a significant role in meeting policy research and engagement responsibilities identified by the Government of Canada – to conduct research in order to remain current with the evolving economy/environment, and to engage with external researchers to produce knowledge and to develop networks that will facilitate discussion of the findings.
  • APRI research and engagement projects are focused on the priorities of the Government of Canada relating to economic development. They support the ACOA Program Activity Architecture, especially in terms of policy decision and direction.

4.1.2 Conclusion - Relevance/Alignment with Government Priorities

APRI is relevant and meeting a demonstrated need for policy research in Atlantic Canada.  Engagement and research activities supported by the program are aligned with the Government of Canada priorities and ACOA areas of interest, as well as the Agency’s policy, advocacy and coordination functions.

4.2 Performance

4.2.1 Incrementality

Within a funding program, incrementality relates to an applicant’s intent and/or ability to proceed with a project – at the proposed location and/or within the proposed timeframe and scope – without government incentive. The level of APRI’s success was assessed against this indicator to demonstrate the extent to which the results being reported are attributable to ACOA’s support.

Client surveys provided a measure of ACOA’s role in facilitating the policy research and engagement activities. The survey results demonstrated that ACOA assistance was highly incremental to the ability of a funding recipient to undertake a project, or to do so with the same scope, quality or timing. Based on survey responses, 97% of funding recipients indicated that without ACOA assistance, the project would have sustained a major negative impact (either not proceeding or proceeding at a much lesser extent); the remaining 3% indicated it would have sustained a minor negative impact.

In an effort to further examine the incrementality of ACOA funding to policy research and engagement activities, respondents were asked whether they would have been able to proceed had ACOA provided less funding. Note that 59% of respondents were recipients of funding for research projects and 41% for engagement projects. As seen in Figure 4, 71% of the research activities would not have proceeded with less funding, compared to 17% for engagement activities. This suggests that the level of funding provided was more incremental for research projects than for engagements. Some engagement activities consisted of supporting travel and attendance at an engagement by an industry representative or an ACOA representative. ACOA representatives attended to either speak on behalf of the Agency or to gather relevant intelligence, and/or to broaden ACOA’s network of partners on a topic of interest. In such cases, the engagement would have proceeded whether or not ACOA sent a representative.

                               Figure 3: Ability to Proceed with Less Funding

 Fifure 3 Ability to Proceed with Less Funding

Half of engagement respondents replied that their projects would have proceeded with less funding from ACOA; however, all stated that less funding would have reduced the scope or affected the quality. For research activities, respondents stated that less funding would have delayed a project’s start, required more time to complete, reduced its scope, or affected its quality.

External experts interviewed were selected based on their understanding and knowledge of sources of funding for policy research and engagements in Canada and the Atlantic region.  These experts emphasized that ACOA plays an important role in supporting the development of high-quality, policy-focused research in the region. Without ACOA’s support, this type of research would be limited.

Considering the perspectives of clients and expert key informants, there is sufficient evidence to state that ACOA’s investment in policy research and engagement activities is having an incremental impact on the results of these activities. 

4.2.2 Achievement of Expected Outcomes

The evaluation explored the achievement of the outcomes outlined in the APRI terms and conditions. 

Outcome 1 Better understanding of the regional economy, strategic sectors or areas for support, and appropriate directions for action in identifying opportunities for economic growth in the Atlantic region.

The majority of G&C files reviewed (89%) contained evidence of providing a better understanding of the Atlantic economy[8]. Survey research respondents (82%) and engagement respondents (58%) reported that their projects provided a better understanding of the Atlantic economy. Key informant interviews supported this finding.

Specific recommendations for policy direction, program direction or further research were found in 60% of the files reviewed. Research (82%) and engagement (58%) survey respondents stated that their projects resulted in specific policy recommendations.

G&C file review indicated that many projects improved our understanding of strategic sectors (44.6%), while 67% of engagement survey respondents and 65% of research respondents stated the same. In cases where the industry is knowledge intensive, industry codes used to construct statistical profiles may not yet exist, and APRI supports the development of alternate methodologies to study these sectors in Atlantic Canada.

Most G&C files reviewed (74.6%) identified areas of support including factors that influence economic development in Atlantic Canada.

It should be noted that of all files reviewed, 96% were found to achieve at least one of the elements of the first intended outcome of APRI.

A key informant expert from the Policy Research Initiative, who has assessed policy research capacity across Canada, commented on ACOA’s ability to produce high-quality policy research with its partners. This research can be used for evidence-based decision making. The informant further elaborated that ACOA is a leader among the RDAs in both quantity and quality of policy research produced.

Outcome 2  Contribution to, or influence on, ACOA’s policy framework and key program activities, as well as areas of interest or priority within those activities, and more coherent process for strategic plan formulation and renewal.

The evaluation focused on the first part of the outcome. That is, this evaluation did not specifically assess the APRI contribution to strategic plan formulation and program renewal; however, key informants mentioned that when preparing for program renewal, ACOA has utilized knowledge produced through the APRI to make a business case. An example of this is the Atlantic Innovation Fund (AIF) renewal, cited as an achievement of Outcome 3. Database analysis, interviews and case studies were utilized to evaluate ACOA’s contribution to the policy framework and key program activities or sub-activities.

As seen in Table 4, database analysis revealed that many research projects, and the greatest amount of funding, were dedicated to economic policy research, or the study of economic issues that impact the Agency’s mandate. Such research projects included studies on the economic impact of universities; productivity and competitiveness (including data purchases from Statistics Canada); issues relating to changing demographics and immigration; and comparative perspectives in regional economic development. Innovation research was also a major focus of APRI, evident by a large number of projects, followed by sector-focused studies and trade and investment research. With respect to funding amounts, economic policy research (35.3%) was followed by sector focus (20.3%), innovation (15.4%), community economic development, entrepreneurship and business skills development and trade and investment.  The coding scheme used by APRI did not allow for the same type of analysis for engagements.

   Table 4: APRI Profile of Research Activities by Type

 

Research Project Type

Number of projects

% of projects

ACOA $ Approved

% of $ Approved

Community Economic Development

4

8.3 221,630 11.1

Economic Policy Research

12

25 702,937 35.3

Entrepreneurship and Skills Development

4

8.3 186,434 9.4

Innovation Policy Research

12

25 307,200 15.4

Sector Focus

9

18.8 404,796 20.3

Trade and Investment

7

14.6 170,817 8.6

Total

48

100 1,993,814 100

 

File review revealed that 72.9% of research or engagements produced knowledge relevant to the PAA program activities or sub-activities. Further analysis revealed that those projects not as closely related to the PAA assessed other aspects of the Atlantic Canadian economy (economic policy research).

Project analysts involved with the case studies strongly believed that the research or engagement projects influenced policy direction or programming relating to ACOA’s program activities and sub-activities (average rating of 8.4 out of 10). In responding to this question, the project analysts provided examples that are explained in further detail in the findings for the Outcome 3. 

Case studies revealed that within Enterprise Development, projects relating to the Atlantic Gateway and cross-border regions supported trade and investment policy and program direction and five case study projects (oceans technology sector, research money conference, Technopolicy conference, innovation capacity building for small and medium-sized enterprises, and economic impact of higher education institutions) supported innovation policy direction and programming. Within Community Development, the Regional Economic Impact Study of Atlantic Canadian Universities enabled ACOA to advocate for the support of these institutions by all levels of government.

Key informants in the area of policy also felt strongly that APRI contributes to both Entrepreneurship Development and Community Development program activities. Examples cited are described in the following section.

Outcome 3  Consideration of recommendations in policy or program design or for further research or event.

G&C files contained evidence of consideration for policy or programs (64%).  Several case studies highlighted the influence of APRI on policies and programs:

  • The Atlantic Gateway project provided research that produced authoritative knowledge and direction for advocacy efforts and coordination with regional stakeholders, including Transport Canada and provincial governments. It also supported the trade and investment program activities relating to the gateway.
  • The oceans technology sector study helped to provide a clearer picture of the industry to assist in advocating for federal and/or provincial support. From a program perspective, it helped to structure innovation support for this sector.
  • The research money conferences raised the profile of research and development in Atlantic Canada in both the public and private sectors. An ACOA vice-president presented at this event on innovation trends, challenges and opportunities using research conducted through APRI. Several Atlantic Canadian leaders in the field have been invited to present at other national research money conferences. The project directly inspired an editorial column calling for the renewal of the AIF.
  • The International Technopolicy Conference, hosted in Halifax, brought global visibility to innovation in Atlantic Canada. Knowledge from expert presentations was utilized in the AIF renewal process. 
  • The study of the economic impact of higher education institutions, a partnership with the Organization for Economic Co-operation and Development (OECD), provided a strong endorsement of ACOA’s approach to innovation through the AIF and the Atlantic Springboard initiative. This outcome highlights APRI’s ability of to produce knowledge that validates the Agency’s policy and program direction in a changing economic environment. Key informants suggested that this was an important role for the initiative.
  • The regional economic impact study of Atlantic Canadian universities produced knowledge that helped ACOA to understand the economic benefits of universities to communities in which they reside – an area of focus within the community development program activity. The project enabled ACOA to federal and provincial governments the importance of universities and their economic benefits to federal and provincial governments.  This included major communications efforts undertaken by the Atlantic Association of Universities such as Ottawa events in which senior-level ACOA representatives participated.
  • The final report on innovation capacity building in small and medium-sized enterprises provided insights for policy and programs, and was referenced in the innovation evaluation and the ongoing corporate innovation and commercialization renewal exercise. Themes and recommendations that emerged from the study have been reflected in new innovation policies and initiatives of provincial departments involved in the project, as reflected in memos to the Agency’s president and minister.
  • The emergence of cross-border regions roundtable and study produced the foundational knowledge on the benefit of trade corridors. The minister communicated the roundtable results to the Atlantic Caucus, including senators, as well as to the Clerk of the Privy Council.  The report contributed to ACOA’s support of the Atlantic Gateway and the Atlantica regional concept in the area of trade and investment.

Key informants from within ACOA’s policy division at head office stated unanimously that APRI had produced knowledge that was considered by the policy and programs branch. With respect to considerations for policy direction, they cited examples such as five policy papers that were produced in 2007; innovation and commercialization; Atlantic Gateway; trade and investment; community development; and ACOA’s current policy direction around trade corridors (the establishment of trade strategies between large centres that benefit communities located along the trade corridor). In speaking more directly about the contribution to ACOA programs, most policy interviewees spoke of the wealth of information produced by the innovation studies that continue to inform the AIF. Studies on the financing of small and medium-sized enterprises inform the enterprise development program activity and contribute to the functioning of the Business Development Program (BDP). Studies on demographic challenges faced by rural areas inform ACOA’s community development program direction. Policy analysts interviewed indicated that all research informs the Agency’s perspective on a given topic by identifying possible areas for new programming or lending support to existing programming. However, interviewees also  cautioned that other factors come to influence whether or not recommendations from APRI research and engagements are implemented, such as changing government priorities, priority of other issues, unexpected findings, and uncovering other important questions to be answered before action can be taken.

ACOA program and regional policy interviewees were less familiar with APRI projects, and generally did not comment on their influence over policies and programs. One interviewee from the programs division had approached APRI over the years with several project ideas, some of which were funded (two projects in foreign direct investment).

Some G&C files were shown to have led to further research (20%). When asked about how the APRI research and engagements have inspired further research, interviewees from policy at head office mentioned that subsequent and related research projects were initiated in the areas of innovation and commercialization, trade corridors, foreign direct investment, demographics and immigration, and the economic impact of higher education institutions. These were verified through the document review.

<Outcome 4</  Integration of findings from engagements or research into policy papers, advice to minister or cabinet.

Twenty percent of files contained evidence of a contribution to policy papers (for more information refer to findings for previous outcomes) or advice to the minister’s office. In a few cases, the minister attended or spoke at the event. Evidence of communications to cabinet was not found in individual project files; however, policy staff stated that knowledge produced by several projects is often pulled together for communications to the minister’s office or to cabinet on a given topic. Such instances were not recorded in individual project files.

Key informants expressed two distinct ways in which the knowledge produced by APRI is disseminated to the minister. On one hand, ACOA proactively communicates information to the minister about a particular study and, on the other hand, the information is shared in response to a query from the minister’s office. A few examples of such queries include the OECD higher education project or data produced relating to forestry or aquaculture. 

Eight case studies were conducted, and it was found in all that projects were used to provide advice to the minister.

Outcome 5 Enhanced policy research capacity in the region that is recognized and used by other stakeholders.

Over the course of the 48 research projects and 40 engagements activities supported through APRI, the funding program has been accessed by 42 different organizations such as universities, think tanks, industry associations and other government partners to conduct or disseminate research. Research funding recipients surveyed indicated that APRI influenced their capacity to conduct research (9.1 out of 10) and they believe the initiative increased policy research capacity in the region (8.1 out of 10). As well, external experts in policy research believe APRI influenced policy research capacity in the region. Taken as a whole, this evidence suggests that the initiative does influence policy research capacity in the region.

Outcome 6  Accumulated knowledge of issues and challenges in Atlantic Canada for use in influencing regional and national positions of federal departments and agencies and other stakeholders, and to prepare better to address Atlantic issues more effectively.

The G&C files contained some evidence that APRI research or engagements had been used to influence the positions of other federal departments and agencies (38%), as well as provincial governments (45%). Survey responses revealed that 75% of engagements and 47% of research were utilized to influence positions of provincial departments, and 67% of engagements and 35% of research were being used to influence positions of federal departments. As such, it appears that were engagement activities were utilized to influence positions of important stakeholders in the region to a greater extent than research activities. Finally, 42% of engagement respondents and 35% of research respondents reported using the funding to influence industry associations. Survey respondents also reported that engagements and research were used to influence other stakeholders such as academics, non-government organizations, the private sector and local governments.

Both external experts in policy research commented on ACOA’s ability to work effectively with key players in Ottawa and in the region on policy issues, and cited examples such as the Public Policy Forum event held in Ottawa, which involved gathering stakeholders and decision-makers to discuss the topic of transformation in the Atlantic Canadian Economy. Another example cited was the cross-border regions study (as previously described).

Case study results were particularly strong in supporting this outcome. Of the eight case studies, seven were used to influence other federal departments and all eight were used to influence provincial departments. For example, the innovation capacity building study influenced the development of provincial innovation initiatives. Also, the Atlantic Gateway project influenced Transport Canada in adopting a policy direction that supports the infrastructure necessary for the gateway. Other groups influenced by APRI in case studies included universities, university and college associations and industry associations.

Outcome 7  Building and maintaining relationships with a more focused and better coordinated network of research partners and stakeholders with common interests.

Files were reviewed for evidence of research partnerships and networking with stakeholders having common interests. Of all G&C files reviewed, 76% contained evidence of funding partners other than ACOA and the recipient organization. Also, 90% of G&C files contained evidence of the involvement of organizations other than funding partners in the project.

Survey respondents were asked to rate the extent to which the project had contributed to building and maintaining a more focused and better coordinated network of research partners. The engagement survey response average was 8.17 out of 10, and the research survey response average was 7.5. Types of research partners cited included universities, colleges, federal and provincial governments, private sector, industry organizations and think tanks. All key informants stated that ACOA has built a considerable network of research partners within and outside Atlantic Canada. A few commented that new talent is continuously emerging in the academic environment and it is important for ACOA policy officers to keep trying to develop that network.

When asked to rate the extent to which the project helped to build and maintain relationships with a more focused and better coordinated network of key stakeholders, the engagement survey response average was 7.83 out of 10 and the research survey response average was 7.25. Types of stakeholders cited included community organizations such as REDOs, municipalities, provincial and federal governments, transfer technology agencies, settlement organizations, industry associations and universities. All key informants from policy at ACOA’s head office commented on the Agency’s ability, through APRI, to develop effective partnerships with other federal departments. For example, ACOA drew upon its network of regional stakeholders to bring together key players for the cross-border regions roundtable, largely contributing to the success of the event. This roundtable was cited by representatives of the national cross-border regions project as the most successful of the regional events. Another example mentioned by key informants was a study with the CD Howe Institute on demographic changes. ACOA was called to support this study in its capacity as an organization with a broad network of stakeholders and partners that could attract key players to the project.

4.2.3 Other Intended or Unintended Outcomes

Key informants within the Agency as well as external experts stated that one unintended outcome of APRI has been it’s ability to connect community stakeholders to a network of other players, or to pertinent information that could assist them in pursuing their own mandates. This in turn, can support ACOA’s mandate.

4.2.4 Key Findings – Performance

Incrementality

• APRI was highly incremental (97% of projects would have sustained a major negative impact without ACOA funds). The provision of lesser funding would have been more detrimental to research than to engagement activities. If ACOA had provided less funding, the majority of research projects would not have proceeded, and engagements would have sustained negative impacts in areas such as quality, scope and completion.

Achievement of Outcomes

  • The research and engagements funded under APRI have contributed to a better understanding of the regional economy, identified areas for support, resulted in specific recommendations, and provided a greater understanding of some strategic sectors. ACOA has been cited by external experts in policy research as being a leader among the RDAs in producing high quality research for evidence-based decision making.
  • The types of research supported by APRI include economic policy research (35.3% of research funds); sector studies (20.4%); innovation research (15.3%); community economic development research (11.1%); entrepreneurship and skills development research (9.4%); and trade and investment research (8.6%). APRI research and engagements have contributed to strategic plan formulation and program renewal, particularly in the area of innovation.
  • APRI has contributed to ACOA’s policy and programs direction, including innovation (e.g. Atlantic Innovation Fund), trade (e.g. Atlantic Gateway), enterprise development (e.g. SME financing), and community development (e.g. impact of higher education institutions). However, ACOA programs staff and regional offices are not very familiar with APRI. While the initiative needs to remain responsive to external economic issues that influence the Agency’s mandate, it also needs to be informed by potential pan-Atlantic policy research topics arising from ACOA programs and regions. APRI management and staff could work more closely with head office programs staff as well as ACOA regional staff.
  • Projects funded under APRI have contributed to policy papers or advice to the minister. Knowledge generated through APRI is utilized to support advice to the minister either proactively (communication of results at the end of a project) or in response to questions originating from the minister’s office.
  • Knowledge produced through APRI is used by ACOA to advocate, particularly during engagement activities. The information resulting from these activities is used by stakeholders to more effectively address Atlantic Canadian issues in regional economic development. External stakeholders include provincial departments, federal departments, industry associations, other non-government organizations, the private sector and local governments.
  • All lines of evidence from this evaluation indicate that APRI has had a pan-Atlantic impact by enhancing policy research capacity in the region and by producing knowledge that is utilized by ACOA and other stakeholders such as provincial governments. APRI has also built and maintained a more focused, better coordinated network of research partners and stakeholders with common interests.

4.2.5 Conclusion - Performance

ACOA plays a key role in producing economic policy research and related engagements in the Atlantic region. The Agency has been successful in meeting APRI intended outcomes. The initiative provides ACOA with the knowledge required to support policy development, advocacy and coordination efforts, and has built a reputation for excellence in policy research. One area identified for improvement would have the APRI management and staff work more closely with ACOA programs and regions to identify research needs.

4.3 Demonstration of Efficiency and Economy

Like many federal departments/agencies, ACOA does not capture costing information related to its operating costs in a manner that would allow for a comprehensive cost-effectiveness or cost-benefit analysis to be conducted for the APRI. In the absence of detailed costing information, alternative measures have been identified to assess the degree to which the initiative is considered to be cost-effective and providing value for money.

4.3.1 Incrementality

The incrementality results identified in Section 4.2 also apply to cost-effectiveness/value for money, as they highlight the value added by ACOA’s involvement in policy research and engagement activities and the necessity of the entire expenditures. As reported, 97% of survey respondents indicated that without ACOA funding, their project would have sustained a major negative impact, including the projects not proceeding. When asked whether the projects would have proceeded with less ACOA funding, 71% of research respondents indicated that they would not have, compared with 17% of engagement respondents (33.3% did not know whether their projects would have been able to proceed). Of all research respondents who would have proceeded with the projects (29%), all reported that less ACOA funding would have resulted in a reduced scope; 42.9% reported that it would have resulted in a delayed start; and 14.3% reported that it would have delayed completion. Of all engagement respondents who would have proceeded with
the less ACOA funds (50%), 72.7% stated that the projects would have sustained decreases in scope and quality, while 18.2% reported that the engagement would not have benefited from the perspective of Atlantic Canada due to lack of representation.

4.3.2 Complementarity/Duplication/Overlap

External experts in policy research and knowledge mobilization, who were selected due to their breadth of knowledge of regional and national funding sources for policy research and engagements, stated that available funding for policy research and engagement activities is limited. Think tanks such as the Atlantic Provinces Economic Council (APEC) and universities do not have sufficient funding to meet policy research needs; however, in working with APRI, these organizations are better able to respond to existing needs. Other national programs that could support Atlantic policy research are funded by the Social Sciences and Humanities Research Council (SSHRC). These include the Community University Research Alliance program, Research Development Initiative, and Northern Communities program. According to external experts, these grants are national and a primary source of academic funding; therefore, the application process is highly competitive and projects have a tendency to lack in policy focus. These
findings suggest that APRI does not duplicate other programs, and while there might be minimal overlap with other research activities, these do not provide the same strategic policy focus. The APRI complements other research conducted by provincial governments and Atlantic Canadian think tanks such as APEC and universities. Also, it complements national research produced by organizations such as the Conference Board of Canada, Statistics Canada, and the Policy Research Initiative, all of which appear as project partners in the list of projects that have taken place over the period 2004-2005 to 2008-2009.

4.3.3 Leveraging of Support from Funding Partners

The results indicate that recipients of APRI funding have been successful at leveraging ACOA’s contributions to obtain additional sources of financial support. ACOA supported 30 research projects, contributing over $1.71 million and leveraging over $1.12 million from other organizations. In more basic terms, for every $1.00 ACOA invested in research, an additional $0.65 was contributed from other organizations. The Agency also supported 25 engagements contributing over $1.18 million and leveraging over $2.4 million from other organizations. For every $1.00 invested by ACOA in engagements, an additional $2.03 was contributed by other organizations. This finding is consistent with findings for incrementality – engagements are able to attract more funding from other sources. 

Overall, ACOA contributed more than $2.89 million to APRI G&C projects and leveraged over $3.52 million from other organizations. Every $1.00 invested in APRI G&C projects resulted in an additional $1.21 contributed from other organizations. According to corresponding data provided in the previous evaluation covering the period 2000-2001 to 2003-2004, leveraging for the APRI G&C projects was $1.01 for every $1.00 invested by ACOA. Current leveraging ability of APRI represents an increase, indicating that the initiative is cost-effective in terms of its leveraging impact.

4.3.4 Alternatives for Design and Delivery or Funding Mechanism

Promotion to Potential Funding Recipients

The evaluation assessed the appropriateness of promoting APRI to potential research and engagement clients outside ACOA. The initiative is not extensively promoted outside the Agency. The ACOA website contains a list of studies that are sponsored under programs such as APRI and provides contact information for inquiries. Most research and engagement clients surveyed have not heard of the APRI through the ACOA website. The majority of engagement (75%) and research (88.2%) respondents identified that they heard about APRI through an ACOA employee. Research respondents also identified colleagues within their own organizations as a source of promotion of APRI (29.4%).

Internal interviewees mentioned that they often come into contact with new policy researchers through engagements such as conferences or by consulting with external colleagues. Six of seven key informants from within the policy division suggested that due limited program resources, promoting APRI too heavily (e.g. higher profile on the Agency website or an extensive call for proposals) could increase the risk of creating too great a demand on limited APRI resources. The increased demand could dilute APRI’s focus, its relevance and its ability to be used strategically by the Agency. However, two of the seven key informants stated that a greater effort on the part of policy analysts is warranted in order to seek new research talent or capacity on an ongoing basis. Canada Economic Development for Regions of Quebec (CED-Q) recently conducted a request for proposals (RFPs) targeted to three pre-determined areas of research in order to grow its network of external policy researchers. The project selection committee for RFPs at CED-Q was comprised of program and policy representatives from within the organization as well as external industry representatives.

Dissemination

In order for APRI to achieve its desired outcomes, it must ensure that the knowledge produced is disseminated appropriately and reaches relevant stakeholders. Every APRI research and engagement activity requires a tailored dissemination strategy to ensure that relevant stakeholders gain from the knowledge produced. This is to say that no single dissemination strategy can be applied to all activities. During the period of 2004-2005 to 2008-2009, 23 research and roundtable reports were posted to the ACOA website. In interpreting this figure, it should be noted that G&C projects produce reports that are not the intellectual property of ACOA and, therefore, cannot always be posted directly to the ACOA website; however, ACOA retains intellectual property rights over the 33 research and engagement activities funded through O&M projects.

G&C files were also examined for evidence of dissemination practices (other than the ACOA website). The reasons for disseminating or not disseminating information were not consistently recorded in the files; therefore, it was not possible to evaluate the appropriateness of these activities. Of all the files, 78% had some evidence of external or internal dissemination (72% external to ACOA and 69% internal to the Agency). Requests for APRI-generated knowledge do not always take place immediately following project completion. When an Agency need arises, research or knowledge generated through APRI is utilized as part of the information required to support decision making. These instances would not be recorded in files.

Survey respondents were also asked about their knowledge of dissemination practices. Responses relating to completed projects revealed that a considerable amount of dissemination work is done on the part of the funding recipient. The majority of respondents for both engagements and research activities reported posting results to their organizations’ websites; 81.8% of engagement respondents reported sharing the report with provincial governments; compared with 64.3% for research respondents. A slightly lower proportion of respondents reported sharing reports with federal departments or agencies, other research institutes, or industry associations. Considerably more engagement respondents (54.5%) reported sharing findings with the media than research respondents (28.6%). Conversely, more research respondents (42.9%) reported presenting findings at a conference than did engagement respondents (27.3%).

From the perspective of key informants from ACOA’s policy division at head office, dissemination is the most important part of producing knowledge. Projects tend to be more widely disseminated when they involve a steering committee comprising various Agency and external representatives. The urgency of a project may, however, prohibit the use of steering committees. A quarterly report summarizing APRI’s ongoing and completed research was circulated widely to ACOA’s policy division at head office, the policy network, vice-presidents and directors, but the practice has not been consistent in recent years. Many interviewees recognized that improvements could be made in this regard, and some policy staff suggested dedicating a full-time employee to manage the communication of knowledge and promotion of research/engagements of the policy division. Possible factors influencing dissemination were cited, such as intellectual property matters, robustness of the findings, and changing government priorities.

Regional policy regional staff and head office program key informants believed that knowledge generated through the APRI does not get disseminated well throughout the Agency. They have noticed the waning of quarterly reports distribution, which they would like to see circulated regularly. Regional policy network members suggest having APRI as a standing item on the Policy Network agenda, with the manager of APRI discussing ongoing projects as well as completed ones. Some of the regional staff has suggested presentations where feasible and appropriate.

Priority Setting

The current process for deciding which policy topics are priorities for APRI has been described as fluid and organic by key informants within policy at head office. Suggested priority influences include proponent proposal ideas, ideas that come from someone in the programs area or the regional offices, the priorities of the Government of Canada, ACOA priorities as identified in documents such as RPPs and the five policy papers, and general shifts in, or concerns with, the economy of Atlantic Canada. All key informants have identified a need to work more closely with others in the Agency to identify research priority topics that will guide project selection. While key informants external to the head office policy division are aware that research on the broader economic context needs to take place, they also believe that a regular (i.e. annual) consultation process with regions and programs representatives could raise pertinent, pan-Atlantic research questions that can inform their work toward the Agency’s mandate. The Policy Network was also suggested as a forum for discussing research ideas and priorities.

Project Selection

In the case of project selection, ideas are brought forward to APRI management; policy managers discuss applications and then make decisions to proceed or not proceed. Several key informants from within and outside the policy division at head office identified a need for a more inclusive decision-making process for project selection. They elaborated that a project selection process with input from only one or two individuals could result in missed opportunities to conduct pertinent research. The authority for project approvals lies with the director general of policy; however, some key informants suggested that proposed projects be deliberated by a committee comprising representatives of policy and other Agency divisions. The committee could recommend projects for selection to the director general. Deliberations could consider proposal quality, and priorities set by the policy area and through consultations.

Performance measurement and monitoring

In order to explore the extent to which performance measurement was undertaken as a part of APRI, the Results-based Management and Accountability Framework (RMAF) constructed in 2005 was reviewed with APRI management and staff. The RMAF contained an ongoing monitoring strategy that included several activities, notably an annual file review, project reporting requirements, annual feedback meetings, an online survey and a capacity check study.

The 2005 RMAF was the first one developed for a policy funding program at ACOA. Feasibility and utility of the requirements had not yet been tested. Generally, most activities identified for the period 2005-2010, other than project monitoring and the current evaluation, were not undertaken, or were conducted to a lesser extent due to their perceived limited value. In recent years, the Agency has constructed a performance measurement framework that guides performance measurement requirements for programs under the PAA. Files were assessed for content relating to the performance measurement strategy. G&C files contained most of the required information, with the exception of dissemination plans. From an audit perspective, O&M files are expected to contain supporting documents for contract rationale, contract documents, and appropriate signatures. Accordingly, activity outputs and outcomes are not required to be documented in the files. A review of a sample of five O&M files found that all contained contracts and three contained documents supporting project rationale. Going forward, performance monitoring will need to include all data elements that are currently collected plus consistent recording of dissemination plans and activities, users of APRI information and project outcomes for both G&C and O&M activities. This will enable the Agency to respond to requirements outlined in its’ performance measurement framework and to prepare for future evaluations. Also, annual quantitative summaries of the APRI activities would help set targets and manage for results.

An examination of the logic model revealed some duplication and overlap in the stated expected outcomes of APRI. These could be further streamlined in order to simplify future accountability exercises. ACOA employees managing APRI have begun this work.

Best Practices and Lessons Learned

The following best practices were identified though key informant interviews and case studies as factors that contributed to the success and impact of a project.

  • Consultation with senior economic analysts and regional stakeholders (internal or external) possessing related knowledge during the plenary stages. This ensured that researchers were employing relevant and valid methodologies, and resulted in slight modifications in the approach, which allowed for the research to better capture targeted data.
  • A partnership approach to developing conference content relevant to knowledge needs of a region (i.e. ACOA programs, provincial governments, ACOA regional offices, project proponents).
  • Strategic partnering with well networked organizations to achieve a wider reach. Project exposure was increased through these organizations’ other activities (e.g. Association of Atlantic Universities).
  • Agency support for research that informs policy and advocacy (e.g. several years ago, ACOA vice presidents were assigned as sector champions and worked with their teams to transform the research into policy development and advocacy work).

Interviews and case studies also brought forward several lessons learned.

  • A communications strategy is required, including distribution of final report to industry representatives and other key stakeholders.
  • Providing additional context about reporting requirements to partners who lack familiarity with the Government of Canada and with requirements of G&C programs at project outset can increase their likelihood of understanding and adhering to the requirements in the end.
  • Obtaining agreement from provincial (and other) partners on expected project deliverables can help to ensure that all parties have a common understanding of project outputs and outcomes.
  • A more inclusive project selection process could increase the likelihood of selecting a broader representation of pertinent projects.

Funding Mechanism

In the interest of streamlining the Agency funding programs and reducing administrative processes associated with having a separate funding program, the evaluation examined whether APRI required its own funding program or could be incorporated into the Agency’s BDP. A review of the BDP terms and conditions revealed that under its business support element, ACOA may make a grant or a contribution to a non-commercial operation for several activities, including “carrying out a development study which is likely to have a significant impact for the development of small and medium-sized business in the region.” According to the terms and conditions of APRI, activities include Agency and stakeholder-initiated research and related activities, roundtables focused on various policy issues and opportunity areas, and policy research conferences. Policy research is carried out on a “broad range of issues, challenges and opportunities associated with the development of the region’s economy.”

Currently, though there may be some overlap between the type of research that can be conducted under APRI and the type of research conducted under the BDP, it is clear that the APRI policy research objectives are broader than what is contained in the BDP, both in terms of the type of research that can be conducted (not solely focused on a likely impact for the development of small and medium-sized enterprises) and in terms of the roundtables or engagements.  Compared to the type of research described in item five of the BDP terms and conditions, much of the policy research conducted under APRI is focused on broad contextual policy issues that influence economic development rather than on SMEs in particular.

From a logistical standpoint, if APRI were to be incorporated into the BDP, the terms and conditions of the BDP would have to be modified to reflect the broader objectives of APRI, and allow for conferences and roundtable engagements to be funded through it. The Agency would also have to plan and fund O&M required for these activities and others such as data purchases and conference participation.

Comments and Concerns

Key informants were asked whether a streamlining of the funding mechanism would increase cost-effectiveness. An interviewee from ACOA’s program division stated that research activities are, can and should be done under the BDP by modifying the terms and conditions. ACOA policy staff from head office and regions cautioned that such an exercise might limit the flexibility of APRI to continue conducting its current activities in the same manner. Most highlighted the differences between the objectives or terms and conditions of the BDP and those of APRI as outlined above. Many wondered what might happen to the O&M funding that is currently an important part of APRI. Some stated that if APRI became part of the BDP, it would be difficult to protect APRI funds from being utilized by competing interests and other urgencies within ACOA’s programs. Some felt that policy research and engagement activities would have all but disappeared had APRI not been established. There was also a concern that future regulation changes within the BDP could produce adverse effects for APRI, which was designed with a different intent. 

4.3.5 APRI Recognized as an Effective Model

Due to the perceived success of APRI over the last decade, another federal regional development agency, CED-Q, approached ACOA to inquire about the APRI model and borrow from it to develop its own G&C funding program for policy research and engagements. An interview was held with the manager of policy at CED-Q. In making their business case and describing value-for-money, the CED-Q officials cited the successes of APRI. One aspect of the CED-Q policy initiative that differs from APRI is that, in some cases, a committee with representatives from within and outside the CED-Q policy area makes decisions about what projects to fund. Another difference is that the Quebec agency has advertised calls for proposals in three key policy areas, in order to uncover policy researchers in its region.

4.3.6 Key Findings – Efficiency and Economy

  • APRI does not duplicate other programs; minimal overlap has been identified with other research activities. The initiative complements other research conducted by provincial governments, universities, and think tanks such as the Atlantic Provinces Economic Council.  It also complements national research produced by organizations such as the Conference Board of Canada, Statistics Canada and the Policy Research Initiative, all of which have established collaborations with APRI.
  • Recipients of ACOA G&C funding through APRI have been successful at obtaining additional sources of financing for research and engagements. Analysis shows that for every dollar of APRI investment, an additional $1.21 is leveraged from other organizations, representing an increase of approximately 16% since the previous APRI evaluation conducted in 2005. These results indicate that APRI is cost-effective in terms of leveraging impact. 
  • The evaluation identified opportunities for improving the effectiveness of program design and delivery of APRI. These included increasing promotion of APRI to new potential research and engagement collaborators; increasing communication of the initiative within the Agency; and establishing more inclusive processes for determining priority areas of research and project selection. Knowledge dissemination plans and activities for individual projects need to be recorded, as well as project outcomes for both G&C and O&M projects. 
  • Best practices identified for future projects include the involvement of key stakeholders in planning for engagements and research, and agreement on project deliverables with other funding partners from the outset.
  • Although some overlap exists between research projects conducted under the BDP and APRI, the BDP’s terms and conditions are not conducive to APRI activities.
  • APRI has been identified as a cost-effective policy research development model, and inspired the CED-Q to develop a business case for a similar policy research G&C funding program.

4.3.7 Conclusion - Efficiency and Economy

APRI is considered to be cost-effective and provide value for money. The evaluation results demonstrate that APRI activities are effective and emphasize the need for further development which could be achieved by increasing promotion to new researchers; enabling more inclusive processes for setting research priorities and selecting projects; and strengthening communications practices and performance measurement of the initiative.

5.0 Conclusions and Recommendations

5.1 Conclusions

5.1.1 Relevance/Alignment with Government Priorities

APRI is relevant and meeting a demonstrated need for policy research in Atlantic Canada.  Engagement and research activities supported by the program are aligned with the Government of Canada’s priorities and ACOA’s areas of interest, as well as the Agency’s policy, advocacy and coordination functions.

5.1.2 Performance

ACOA plays a key role in producing economic policy research and related engagements in the Atlantic region. The Agency has been successful in meeting APRI intended outcomes. The initiative provides ACOA with the knowledge required to support policy development, advocacy and coordination efforts, and has built a reputation for excellence in policy research. One area identified for improvement would have the APRI management and staff work more closely with ACOA programs and regions to identify research needs.

5.1.3 Efficiency and Economy

APRI is considered to be cost-effective and to provide value for money. The evaluation results demonstrate that APRI activities are effective and emphasize the need for further development which could be achieved by increasing promotion to new researchers; enabling more inclusive processes for setting research priorities and selecting projects; and strengthening communications practices and performance measurement of the initiative.

5.2 Recommendations

This evaluation has identified opportunities for improvement, leading to the following recommendations to further the achievement of desired APRI outcomes.
 

  1. Implement more inclusive processes in setting priorities and selecting projects.
  2. Develop internal and external communications plans in order to promote APRI and disseminate knowledge.
  3. Keep an organized account of data on O&M projects, as these are subject to accountability requirements, along with the G&C projects to strengthen performance measurement. The data recorded should systematically include the rationale behind the dissemination strategy and parties to which the report was disseminated, irrespective of the medium (mail, e-mail, web links). Data on project outcomes should also be recorded.
  4. Foster the application of the following best practices identified in APRI delivery:
    1. the involvement of key regional stakeholders in planning for engagement and research to increase regional relevance; and
    2. the definition of project deliverables with funding partners from the outset.

6.0 Alignment of Key Findings, Conclusions and Recommendations

Key findings, associated, conclusions and recommendations are presented in Figure 4.

  Figure 4: Alignment of Conclusions, Key Findings and Recommendations

 

 Figure 4 Alignment of Conclusions

 

Appendix A: Methodology

The APRI evaluation approach utilized multiple lines of evidence supported by a mix of both qualitative and quantitative methods. Findings from each line of enquiry were compared using a triangulation approach to assess consistency. During this analysis, strength of the methods, supporting evidence and possible biases were considered in the determination of key findings.

Prior to undertaking the research, preliminary consultations were conducted with ACOA officials to ensure a comprehensive evaluation design.  These consultations included a meeting with the Agency’s APRI staff to discuss scope, timeline and evaluation issues; a review of policy background documents; an examination of ACOA administrative data; and the circulation of the evaluation terms of reference (including evaluation issues) to senior ACOA staff in both head office and the regions. Information gathered was utilized to finalize evaluation issues and inform the methodology.

APRI officials were also consulted in planning the methodology and in developing the list of key informant interviewees and case studies. Based on this initial consultation, refinements were made to the evaluation matrix, including the evaluation questions and corresponding indicators, data sources and methods.

The evaluation included six targeted research approaches as detailed below.

Document and Literature Review

A literature review of existing documents was undertaken to assess the relevance and success of APRI.  Literature was also reviewed to assess alternatives and best practices from policy research and engagement programming in other regional development agencies.

The following types of documents were analyzed during the evaluation:

  • general background APRI documentation (e.g. Treasury Board submissions, RMAFs, terms and conditions, APRI quarterly report, previous APRI evaluation, Departmental Performance Reports, ACOA website);
  • documentation on the role of policy within the government of Canada, policy research priorities of the government of Canada and of ACOA (e.g. PRI report and speech from the Clerk of the Privy Council, ACOA Reports on Plans and Priorities, ACOA policy papers); and
  • documentation on other approaches for policy research and engagements (e.g. other RDA evaluations, terms and conditions, business plan presentation, BDP terms and conditions).

A bibliography is included as Appendix B.

Administrative Data Analysis

The following data from ACOA administrative databases were analyzed:

  • QAccess data on all 55 grants and contributions (G&C) projects approved between 2004-2005 and 2008-2009; and
  • Administrative records on 35 operation and maintenance (O&M) contracts kept by staff managing APRI.

G&C and O&M File Review

Fifty five G&C and 35 O&M paper files were obtained from ACOA central records and analyzed against evaluation issues. The following key documents were reviewed:

  • project proposals were analyzed for rationale, linkage to ACOA and government priorities identification of intended participants, partners, and funding sources;
  • project summary forms were analyzed for relevance to ACOA and APRI, funding details and funding partners;
  • final research or conference reports were analyzed for results, participation, recommendations and content addressing the objectives and criteria of APRI; and
  • communications documents were analyzed for evidence of decision-making practices with regard to project selection and dissemination of knowledge produced (e.g. emails, briefings, memos and presentations).

Key Informant Interviews

Key informant interviews were conducted to gather evidence on most of the evaluation issues. The questionnaires were reviewed/validated by program staff prior to finalization, and were pre-tested by the project team during the initial interviews to confirm their validity. A total of 16 key informant interviews were conducted (Table 5).

                                     Table 5: Summary of Key Informants

 

Type of Respondents

Number

ACOA head office Senior Personnel (DG and VP level)

3

APRI Management and Staff

4

External Experts

2

Colleague from another RDA

1

Internal Users

6

Total

16

Combinations of telephone or in person interviews were conducted.  All individuals contacted for interviews participated, for a 100% participation rate.

Surveys

Two short internet surveys were designed and conducted, one with APRI research funding recipients and the other with APRI engagement funding recipients. These were to assess client perceptions of APRI relevance (including incrementality) and promotion, and its APRI outcomes. Survey questionnaires were reviewed and validated by APRI staff.
In order to reduce response burden, funding recipients were only asked to respond to one survey, even if they had participated in many projects.  Only external funding recipients were surveyed. Surveys were sent to 35 research funding recipients and 21 engagement funding recipients. Initially, response rates were low. Recipients were then contacted to ensure they had received the survey and it was discovered that 5 respondents had moved to other organizations and were not able to be contacted. Response rates were 53.5% for research funding recipients and 63% for engagement funding recipients, which corresponds to expected rates for internet surveys.

Case Studies

A thematically representative and small sample of case studies were identified to permit a more in-depth analysis of funded projects and their impacts. Eight case studies were selected to illustrate best practices. Case studies involved a review of project documents and interviews or information exchange with project managers.  The detail explored through the case studies was restricted to ACOA staff due to the scope of the evaluation, which was adjusted to correspond to the materiality and risk of APRI.

Selection of case studies was informed by the distribution of APRI projects by type (ACOA assistance and #). The list includes both O&M and G&C projects.

The list of case studies is as follows.

Research Projects

  • Atlantic Gateway
  • OECD Study on the Impact of Post-Secondary Education Institutions
  • Value of the Ocean Technology Sector
  • Regional Economic Impact Study of Atlantic Canadian Universities
  • MInnovation Capacity Building in SMEs

Engagements

  • Technopolicy Conferences
  • Emergence of Cross-Border Regions
  • Research Money Conferences

Appendix B: Bibliography

Government of Canada, ACOA. 2007. Policy Priorities for ACOA: Summary of Analysis and Proposals for Future Directions.

Government of Canada, ACOA. “Report on Plans and Priorities 2008-2009” http://www.tbs-sct.gc.ca/rpp/2008-2009/inst/aco/aco00-eng.asp.  Accessed November 2009.

Government of Canada, ACOA. “Report on Plans and Priorities 2007-2008” http://www.tbs-sct.gc.ca/rpp/2007-2008/ACOA-APECA/ACOA-APECA00-eng.asp . Accessed November 2009.

Government of Canada, ACOA. “Report on Plans and Priorities 2006-2007” http://www.tbs-sct.gc.ca/rpp/2006-2007/ACOA-APECA/ACOA-APECA00-eng.asp.  Accessed November 2009.

Government of Canada, ACOA. “Report on Plans and Priorities 2005-2006” http://www.collectionscanada.gc.ca/webarchives/20060118031129/http://www.tbs-sct.gc.ca/est-pre/20052006/acoa-apeca/acoa-apecar56_e.asp.  Accessed November, 2009.

Government of Canada, ACOA. “Report on Plans and Priorities 2004-2005.” http://www.collectionscanada.gc.ca/webarchives/20060117214436/http://www.tbs-sct.gc.ca/est-pre/20042005/acoa-apeca/acoa-apecar45_e.asp.  Accessed November 2009.

Government of Canada, Clerk of the Privy Council, “Policy Making in the 21st Century: New Challenges for Canada” http://www.pco-bcp.gc.ca/index.asp?lang=eng&Page=clerk-greffier&Sub=speeches-discours&Doc=20091013-eng.htm.  Accessed November 2009.

Government of Canada, Department of Finance. 2006. “Advantage Canada: Building a Strong Economy for Canadians” http://www.fin.gc.ca/ec2006/plan/pltoc-eng.asp.  Accessed November 2009.

Government of Canada, Policy Research Initiative. 2009. “Capacity, Collaboration and Culture:  The Future of the Policy Research Function in the Government of Canada” http://www.policyresearch.gc.ca/page.asp?pagenm=2009-0010_toc.  Accessed November 2009.

Footnotes 

1  Government of Canada. WED. Evaluation of Policy Advocacy and Coordination. (2009)
2  Government of Canada. ACOA. APRI Terms and Conditions (2005)
3  Government of Canada. ACOA. ACOA Program Activity Architecture, 2007-2008 and 2008-2009
4  Government of Canada. ACOA. ACOA Program Activity Architecture, 2007-2008 and 2008-2009
5  This figure includes a comparative study of health and safety and environment standards valued at  $45,181.15. The study was coded into GX using coding unfamiliar to current APRI staff and, as such, was not captured as part of the evaluation design.
6  Government of Canada. Policy Research Initiative. Capacity, Collaboration and Culture: The Future of the Policy Research Function in the Government of Canada, page 5 (March 2009).
7  Government of Canada. Clerk of the Privy Council. “Policy Making in the 21st Century: New Challenges for Canada”, page 4 (October 2009).
8  O&M files did not contain sufficient information (i.e. final reports and correspondence to the extent of G&C files) to conclude on findings.