BEALS, LALONDE & ASSOCIATES
Determinations
BEALS, LALONDE & ASSOCIATES
v.
DEPARTMENT OF JUSTICE
File No. PR-2004-009
Determination and reasons issued
Tuesday, July 27, 2004
Corrigendum issued
Tuesday, September 14, 2004
TABLE OF CONTENTS
IN THE MATTER OF a complaint filed by Beals, Lalonde & Associates under subsection 30.11(1) of the Canadian International Trade Tribunal Act, R.S.C. 1985 (4th Supp.), c. 47;
AND FURTHER TO a decision to conduct an inquiry into the complaint under subsection 30.13(1) of the Canadian International Trade Tribunal Act.
| BEALS, LALONDE & ASSOCIATES |
Complainant |
| AND |
|
| THE DEPARTMENT OF JUSTICE |
Government Institution |
DETERMINATION OF THE TRIBUNAL
Pursuant to subsection 30.14(2) of the Canadian International Trade Tribunal Act, the Canadian International Trade Tribunal determines that the complaint is valid.
Pursuant to section 30.16 of the Canadian International Trade Tribunal Act, the Canadian International Trade Tribunal awards Beals, Lalonde & Associates its reasonable costs incurred in preparing and proceeding with the complaint, which costs are to be paid by the Department of Justice. The Canadian International Trade Tribunal's preliminary indication of the level of complexity for this complaint case is Level 1, and its preliminary indication of the amount of the cost award is $1,000. If any party disagrees with the preliminary indication of the level of complexity or the preliminary indication of the amount of the cost award, it may make submissions to the Canadian International Trade Tribunal as contemplated by the Guideline for Fixing Costs in Procurement Complaint Proceedings. The Canadian International Trade Tribunal retains jurisdiction to establish the final amount of the award.
James A. Ogilvy
James A. Ogilvy
Presiding Member
Susanne Grimes
Susanne Grimes
Acting Secretary
| Tribunal Member: |
James A. Ogilvy, Presiding Member |
| Investigation Officer: |
Michael W. Morden |
| Counsel for the Tribunal: |
Nick Covelli |
| Complainant: |
Beals, Lalonde & Associates |
| Government Institution: |
Department of Justice |
| Counsel for the Government Institution: |
Derek Rasmussen |
Please address all communications to:
The Secretary
Canadian International Trade Tribunal
Standard Life Centre
333 Laurier Avenue West
15th Floor
Ottawa, Ontario
K1A 0G7
Telephone: (613) 993-3595
Fax: (613) 990-2439
E-mail:
STATEMENT OF REASONS
COMPLAINT
1. On April 28, 2004, Beals, Lalonde & Associates (Beals) filed a complaint with the Canadian International Trade Tribunal (the Tribunal) under subsection 30.11(1) of the Canadian International Trade Tribunal Act.1 The complaint concerned the procurement (Solicitation No. 1000004414) by the Department of Justice (DOJ) for a summative evaluation of the DOJ's Youth Justice Renewal Initiative.
2. Beals alleged that its proposal was not evaluated in accordance with the criteria contained in the Request for Proposal (RFP). More specifically, it alleged that the rating grid used by the evaluation team introduced rating criteria not included in the RFP. It requested compensation for the time and effort that it expended in preparing its proposal. It also requested that a directive be issued to all contracting officers that tender documents must clearly identify the criteria to be used in the evaluation of bids and the methods of weighting and evaluating the criteria, as required by the Agreement on Internal Trade.2
3. On May 6, 2004, the Tribunal informed the parties that the complaint had been accepted for inquiry, as it met the requirements of subsection 30.11(2) of the CITT Act and the conditions set out in subsection 7(1) of the Canadian International Trade Tribunal Procurement Inquiry Regulations.3 On May 31, 2004, the DOJ filed a Government Institution Report (GIR) with the Tribunal. On June 10, 2004, Beals filed its comments on the GIR. On June 23, 2004, the Tribunal requested additional information from the DOJ, which it submitted to the Tribunal on June 25, 2004. Specifically, the Tribunal asked the DOJ:
Please provide a more detailed explanation of the Contractor Selection Method described in paragraph 8.4 of the RFP. Specifically, how were points allocated for the financial proposals and how was the final score determined? What were the proposed prices and overall scores (broken out into both the technical and financial components) of all bidders who successfully passed both the mandatory criteria and the minimum required score for the rated criteria?
4. The public version of the DOJ's additional information was provided to Beals for comment on June 30, 2004. Beals did not submit any comments.
5. Given that there was sufficient information on the record to determine the validity of the complaint, the Tribunal decided that a hearing was not required and disposed of the complaint on the basis of the information on the record.
PROCUREMENT PROCESS
6. The RFP was issued by the DOJ on February 16, 2004, with a closing date for the receipt of bids of March 29, 2004.
7. The RFP read in part as follows:
3.4 Description and Scope of the Work:
The Contractor will perform a summative evaluation of the Youth Justice Renewal Initiative (YJRI), which will assess the relevance, success, and cost-effectiveness of YJRI.
3.4.1 Tasks
The Contractor shall:
a) Develop a clear and sound methodology that is based on the triangulation of multiple lines of evidence, both qualitative and quantitative, and addresses the study issues and questions identified below;
b) Develop data collection instruments and, where relevant, sampling methods for each line of inquiry;
c) Develop and populate a matrix of methods and results that identifies, for each evaluation question, the indicators, the data sources and methods, and the results;
d) Prepare a work plan and research design report;
e) Conduct the data collection;
f) Prepare a detailed point-form outline of the evaluation report;
g) Prepare a draft evaluation report; and
h) Prepare a final evaluation report.
3.4.4 Methods
8. The RMAF [Results-based Management and Accountability Framework] identifies possible methods for this evaluation. Proposers may suggest additional or alternative methods.
8.2.1 Experience of Team Members in conducting evaluations (maximum of 15 points)
8.2.2 Experience of Team Members in conducting research or evaluation in the area of youth justice (maximum of 15 points)
8.2.3 Understanding of the Youth Justice Renewal Initiative (maximum of 20 points)
8.2.4 Appropriateness of methodology (maximum of 50 points)
A minimum of 70 points shall be required for the proposal to be judged valid and responsive.
8.4 Contractor Selection Method
All proposals will be rated on technical acceptability before the financial proposal is considered. The best-value-for-money compliant proposal will be chosen, using a 80/20 ratio of technical points/cost. This means that the lowest overall cost proposal will not necessarily be chosen.
9. There were five questions submitted to the DOJ during the bidding period. These questions, and the answers thereto, were made available to all bidders. Question and answer No. 1, which relate to this complaint, read as follows:
Q.1 a) Mandatory Requirement 8.1.4 specifies that "Bidders must demonstrate their experience in the conduct of at least 2 evaluation studies similar in scope and complexity to the project described in this RFP." Does this mean only that we must demonstrate that we have undertaken projects featuring the same type and complexity of tasks as those described in section 3.4 of the proposal, or does it also imply that we must demonstrate that we've undertaken projects with a similar dollar value?
b) If scope does include dollar value, can you indicate how important a part of the criteria that is, and what dollar value would be seen as similar? Is 90K similar? Is 75K?
A.1 `Similar in Scope and Complexity' means - You have to demonstrate that you have undertaken projects featuring the same type and complexity of tasks as those described in section 3.4 of the proposal and it also impl[ies] that you must demonstrate that you've undertaken projects with a similar dollar value of at least $60,000 and that the project lasted more than 3 months.
10. According to the DOJ, seven proposals were received, all of which were judged to have met the mandatory criteria set out in section 8.1 of the RFP. The proposals were independently reviewed by four evaluators, who met on April 6, 2004, to determine the consensus scores to be awarded to each of the bidders on the rated criteria of their respective proposals. The evaluators were provided with a rating grid to assist them in their review of the proposals. Regarding the rated criteria, the rating grid provided in part:
| Rated Requirements |
|
| Experience of Team members in conducting evaluations similar in "scope and complexity"3 to the project described in the RFP: Poor/2 comparable projects: 1-4 points Comments: |
|
| Experience of Team members in conducting research or evaluation in the area of youth justice: Poor/0-2 projects: 1-4 points Comments: |
|
| Understanding of the Youth Justice Renewal Initiative: Points that would demonstrate understanding of the YJRI would include: - understanding of the objectives of the YJRI Poor/does not demonstrate understanding of any of the above points: 1-5 points Comments: |
|
| Appropriateness of the Methodology Criteria that should be considered in the rating of the appropriateness of the methodology include: Sufficiency of the number of data sources Poor/methodology meets none of the above criteria: 1-14 points |
|
| Total |
/100 |
3 Similar in scope and complexity means an evaluation that has a budget of over 60K, lasted at least 3 months and involved similar methodology, including a similar number of respondents and respondent groups. Note that the Project Authority will contact the two references to obtain information on the projects identified pursuant to Section 8.1.4.
11. On April 14, 2004, the DOJ awarded the contract to Moyer & Associates and sent letters informing the unsuccessful bidders of the contract award. Beals requested a debriefing, which was provided by the DOJ on April 19, 2004. After the debriefing, Beals was provided with a copy of the rating grid.
12. Beals submitted its complaint to the Tribunal on April 28, 2004.
POSITIONS OF THE PARTIES
DOJ's Position
13. The DOJ submitted that the content of the rating grid with respect to the rated requirements is consistent with the rated requirements set out in section 8.2 of the RFP and rationally connected to them. It submitted that the bidders should reasonably have been able to anticipate the subcriteria set out in the rating grid.
14. Regarding rated criterion 8.2.1-"Experience of Team Members in conducting evaluations"-the DOJ submitted that its response to question No. 1 regarding mandatory criterion 8.1.4 makes reference to a similarity in methodologies between projects. It submitted that, therefore, in order for a bidder to receive further consideration as against the rated requirements, it was necessary for the bidder to demonstrate experience in conducting evaluation studies similar in scope and complexity. It also submitted that it necessarily follows that a reasonable bidder would understand that, when describing additional experience in the context of rated criterion 8.2.1, projects similar in scope and complexity should be listed.
15. Regarding rated criterion 8.2.2-"Experience of Team Members in conducting research or evaluation in the area of youth justice"-the DOJ acknowledged that the initial evaluation did not take into account the experience of one of Beals's resources. It submitted that, even if Beals had been awarded more points as a result of this oversight, it would not have been sufficient to alter the bidder's final standing. It also submitted that, despite Beals's allegation that the RFP contained no information to allow a bidder to appreciate the importance, and significance to the evaluators, of the role to be played by team members who possessed significant experience in the area of youth justice, any reasonable bidder would have understood that a proposal that allocated significant aspects of the work to individuals with more extensive and substantive experience in the area of youth justice would receive higher scores during the evaluation.
16. Regarding rated criterion 8.2.3-"Understanding of the Youth Justice Renewal Initiative"-the DOJ submitted that the four subcriteria, provided as part of the rating grid, should have reasonably been anticipated by prospective bidders. It also submitted that the score awarded to Beals reflected the fact that the evaluators had accepted that the Beals team had a good understanding of the YJRI.
17. Regarding rated criterion 8.2.4-"Appropriateness of methodology"-the DOJ submitted that the criteria mentioned in the rating grid naturally flow from an assessment of the appropriateness of any methodology submitted and certainly should have been foreseen by any prospective bidder. It submitted that they are reasonable and rationally connected to the rated criterion.
18. The DOJ submitted that the Tribunal has previously determined that the introduction of subdivisions of criteria and weighting set out in the RFP is entirely appropriate. It submitted that in one of its decisions,4 the Tribunal had accepted this practice as long as the subcriteria and weighting could readily have been anticipated by the bidders and derived from the broader criteria and weighting set out in the RFP. In that case, the Tribunal also stated that the subcriteria could not augment or diminish the relative worth and importance of the criteria announced in the RFP. The DOJ submitted that the rating grid provided to evaluators follows this direction.
19. The DOJ requested that the complaint be dismissed. In addition, it submitted that Beals's requested relief of $20,000 was an arbitrary figure that was unsupported by evidence. It submitted that, if the Tribunal chose to award any amount to Beals, that amount must be significantly discounted so as to reflect the inherent risk that Beals's proposal would not have been the one selected. The DOJ argued that five other bidders, in addition to Beals and the contract awardee, had submitted proposals that complied with the mandatory requirements of the RFP.
Beals's Position
20. Beals submitted that it does not agree with the DOJ's contention that the rating grid was appropriate for an assessment of the rated requirements. It submitted that the criteria in the RFP defined general issues, but provided little or no guidance as to the methods of weighting and evaluating the criteria. Beals submitted that, since the DOJ admitted to developing the rating grid after the closing date for the receipt of bids,5 it indicated that the DOJ was not clear on how it would evaluate the proposals when it posted the RFP. Beals also submitted that it was verbally advised on April 19, 2004, by the DOJ that five of the seven proposals were judged not to have met the threshold of 70 points required for the rated criteria. Beals submitted that this fact indicates that the bidders had insufficient information on which to develop their bids.
21. Beals submitted that the Tribunal's decision in Siemens Westinghouse, referenced by the DOJ, is based on facts that are significantly different from those of the current complaint case. Beals submitted that, in Siemens Westinghouse, the RFP was quite detailed as to the subdivisions of criteria and weighting and what the Crown expected bidders to address. In the present case, Beals submitted, the only breakdown in the RFP was that four criteria (rated criteria 8.2.1, 8.2.2, 8.2.3 and 8.2.4) would be assessed, without any subdivisions, such as those identified under "Understanding of the Youth Justice Renewal Initiative" and "Appropriateness of the Methodology" in the rating grid.
22. Regarding rated criterion 8.2.1-"Experience of Team Members in conducting evaluations"-Beals submitted that the DOJ's answer to question No. 1, as detailed above, related to the mandatory requirements only and that no reference was made to any of the rated requirements, specifically requirement 8.2.1. Beals submitted that it was not reasonable for bidders to have to guess that, in this instance, the response meant that the phrase "conducting evaluations" in rated criterion 8.2.1 meant "conducting evaluations similar in scope and complexity". Beals submitted that it should have received a higher mark based on its experience in meeting the requirement of "conducting evaluations" as stated in the RFP.
23. Regarding rated criterion 8.2.2-"Experience of Team Members in conducting research or evaluation in the area of youth justice"-Beals argued that it had used the same bidding strategy in previous competitions, which consisted of expert advisors being allocated a small percentage of the days and having the majority of the days provided by team members with extensive evaluation experience. It argued that there was nothing in the RFP to indicate that the same strategy would not be successful in this solicitation.
24. Regarding rated criterion 8.2.3-"Understanding of the Youth Justice Renewal Initiative"-Beals argued that the listing of subcriteria was not exhaustive and that it was not reasonable for the DOJ to have expected bidders to select the specific issues identified in the rating grid instead of other equally applicable possibilities.
25. Regarding rated criterion 8.2.4-"Appropriateness of methodology"-Beals submitted that, during its debriefing, the contracting authority informed Beals that it had, among other things, "failed to display creativity in its methodology" and that the contracting authority had noted that section 3.4.4 of the RFP, which stated, "[p]roposers may suggest additional or alternative methods", provided the evaluation team with the justification to deduct points from Beals's proposal. Beals argued that the RFP did not indicate that diverging from the methods outlined in the Results-based Management and Accountability Framework would be an asset or that things like the number of person-days would be a criterion. Beals argued that, given that criterion 8.2.4 was worth 50 percent of the mark, bidders should have received more information or the evaluation should have addressed more than just the five stated subcriteria.
26. In its comments on the GIR, Beals provided the Tribunal with a description of the work performed by its employees and other resources to plan, staff, coordinate and write the proposal, to obtain and assess feedback on its proposal, and to prepare the complaint itself. It claimed that $20,000 represented the cost associated with the level of effort required to perform all the above-listed tasks.
TRIBUNAL'S DECISION
27. Subsection 30.14(1) of the CITT Act requires that, in conducting an inquiry, the Tribunal limit its considerations to the subject matter of the complaint. Furthermore, at the conclusion of the inquiry, the Tribunal must determine whether the complaint is valid on the basis of whether the procedures and other requirements prescribed in respect of the designated contract have been observed. Section 11 of the Regulations further provides that the Tribunal is required to determine whether the procurement was conducted in accordance with the applicable trade agreements, which, in this case, is the AIT.
28. Article 506(6) of the AIT reads in part as follows:
The tender documents shall clearly identify the requirements of the procurement, the criteria that will be used in the evaluation of bids and the methods of weighting and evaluating the criteria.
29. The issue before the Tribunal is whether the evaluation criteria set out in the rating grid were the same as, or could have been predicted from, the evaluation criteria contained in the RFP. In other words, did the tender documents clearly identify the criteria that were used in the evaluation of bids and the methods of weighting and evaluating the rated requirements? With the exception of rated criterion 8.2.1, the Tribunal finds that the rating grid is consistent with the principle laid down in Siemens Westinghouse, as the subcriteria and weighting could have been anticipated by the bidders and the subcriteria did not augment or diminish the relative worth and importance of the criteria in the RFP. The Tribunal finds that the rating grid for these criteria did not deviate from what the Tribunal would reasonably expect a bidder to anticipate regarding each criterion. In the case of rated criterion 8.2.1, the Tribunal finds that the rating grid provided additional information that was not evident when compared to the contents of the RFP. Accordingly, the Tribunal finds that the complaint is valid.
30. Regarding rated criterion 8.2.1-"Experience of Team Members in conducting evaluations"-the Tribunal finds that the addition of the words "similar in `scope and complexity' to the project described in the RFP" [footnote omitted] in the rating grid could not have been reasonably anticipated by reading the RFP. The additional information, had it been known to bidders, would almost certainly have resulted in the DOJ receiving different bids from certain bidders, including Beals.
31. Regarding rated criterion 8.2.2-"Experience of Team Members in conducting research or evaluation in the area of youth justice"-the Tribunal finds that the additional information provided in the rating grid does not exceed what a reasonable bidder would expect in reading the RFP. The Tribunal notes however that the DOJ has acknowledged that its initial assessment of the experience of one of Beals's proposed resources was incorrect and that Beals may have been entitled to additional points.
32. Regarding rated criterion 8.2.3-"Understanding of the Youth Justice Renewal Initiative"-the Tribunal is of the opinion that the additional information contained in the rating grid does not exceed what should reasonably have been expected by the bidders. The Tribunal notes however that this section of the rating grid states: "Points that would demonstrate understanding of the YJRI would include". In the Tribunal's interpretation, this phrase does not mean that the list of points was meant to be exhaustive, as claimed by Beals.
33. Regarding rated criterion 8.2.4-"Appropriateness of methodology"-the Tribunal finds that the subcriteria included in the rating grid could reasonably be expected, given the nature of the requirement and the contents of the RFP. In the Tribunal's opinion, it is not unreasonable to expect bidders to address items such as the number of interviews, types of data sources, etc., as well as to provide adequate information to demonstrate that the methodology that they were proposing to conduct the summative evaluation would produce accurate results. The Tribunal notes that the DOJ has not denied that Beals was informed during the debriefing that creativity was rated. However, having examined carefully the comments of each evaluator on the rating grid with respect to "Appropriateness of the Methodology" proposed by Beals, the Tribunal sees no evidence that creativity was indeed rated. Therefore, the Tribunal finds that, on a preponderance of evidence, creativity was not a subcriterion and was not rated.
34. The Tribunal notes Beals's argument that criterion 8.2.4, given its relative value, should have been better explained in the RFP. The Tribunal cannot consider that argument, as the period for accepting a complaint on this basis has expired. Any concerns about the structure and/or content of the RFP would have to have been submitted to the Tribunal within 10 working days of the discovery of the alleged problem. This time frame would have ended 10 working days after Beals became aware or should reasonably have become aware of the grounds of complaint. Beals should have been aware of this ground when it read the RFP, in other words, no later than the day on which the proposals were due to the DOJ. Any complaints regarding the RFP would therefore have to have been submitted by April 12, 2004. The complaint was submitted on April 28, 2004.
35. In determining an appropriate remedy, the Tribunal considered all the circumstances relevant to this procurement, including those outlined in subsection 30.15(3) of the CITT Act.
36. In the Tribunal's view, although it was a serious deficiency in the procurement process to use a rating grid where portions could not be predicted from the wording of the original RFP, the rating information provided by the DOJ demonstrated that, even if Beals had been awarded full marks in rated criterion 8.2.1 (where the Tribunal found the rating grid to be improper) and rated criterion 8.2.2 (where the DOJ acknowledged that it had not rated Beals's proposal properly), the contractor selection method described at section 8.4 of the RFP would still have resulted in the contract being awarded to the same company. Hence, the Tribunal finds that Beals has not suffered prejudice as a result of the DOJ's breaches of the trade agreements.
37. Regarding Beals's requested relief of $20,000, the Tribunal believes that the costs relating to the time and effort of preparing proposals and attending debriefings should be borne by Beals, as these costs are incurred in the normal course of business. In other words, Beals would have incurred these costs even if its proposal had been evaluated properly, whether or not its bid had been successful.
38. This type of deficiency, if repeated or more extensive, could prejudice the integrity and efficiency of the competitive procurement process. However, in this case, the Tribunal's analysis indicates that the end result would have been the same regardless of the violation and that, therefore, the prejudice to the integrity and efficiency of the competitive procurement process was minimal. The evidence does not indicate that the DOJ was acting in bad faith. Consequently, the Tribunal will not recommend a remedy in this case.
39. The Tribunal will award Beals its reasonable costs incurred in preparing and proceeding with the complaint. The Tribunal has considered its Guideline for Fixing Costs in Procurement Complaint Proceedings (Guideline) and is of the view that this complaint case has a complexity level corresponding to the lowest level of complexity referred to in Appendix A of the Guideline (Level 1). The Guideline contemplates classification of the level of complexity of complaint cases based on three criteria: the complexity of the procurement; the complexity of the complaint; and the complexity of the complaint proceedings. The complexity of the procurement was medium, in that it involved services relating to a defined study. The complexity of the complaint was low, in that it involved a single issue and a single aspect of a single trade agreement. Finally, the complexity of the complaint proceedings was low, as there were no interveners, no motions and no public hearing, and the 90-day time frame was respected. Accordingly, as contemplated by the Guideline, the Tribunal's preliminary indication of the amount of the cost award is $1,000.
40. Beals had requested that the Tribunal issue a directive to all contracting officers that tender documents must clearly identify the criteria that will be used in the evaluation of bids and the methods of weighting and evaluating the criteria. The Tribunal believes that the AIT clearly indicates this requirement and that this decision should serve as sufficient notice to the DOJ to that effect.
DETERMINATION OF THE TRIBUNAL
41. Pursuant to subsection 30.14(2) of the CITT Act, the Tribunal determines that the complaint is valid.
42. Pursuant to section 30.16 of the CITT Act, the Tribunal awards Beals its reasonable costs incurred in preparing and proceeding with the complaint, which costs are to be paid by the DOJ. The Tribunal's preliminary indication of the level of complexity for this complaint case is Level 1, and its preliminary indication of the amount of the cost award is $1,000. If any party disagrees with the preliminary indication of the level of complexity or the preliminary indication of the amount of the cost award, it may make submissions to the Tribunal, as contemplated by the Guideline. The Tribunal retains jurisdiction to establish the final amount of the award.
IN THE MATTER OF a complaint filed by Beals, Lalonde & Associates under subsection 30.11(1) of the Canadian International Trade Tribunal Act, R.S.C. 1985 (4th Supp.), c. 47;
AND FURTHER TO a decision to conduct an inquiry into the complaint under subsection 30.13(1) of the Canadian International Trade Tribunal Act.
| BEALS, LALONDE & ASSOCIATES |
Complainant |
| AND |
|
| THE DEPARTMENT OF JUSTICE |
Government Institution |
CORRIGENDUM
In the English version of the Statement of Reasons, the second subparagraph of the RFP quotation should read as follows:
3.4.4 Methods
The RMAF [Results-based Management and Accountability Framework] identifies possible methods for this evaluation. Proposers may suggest additional or alternative methods.
In paragraph 8, the first two sentences, "The RMAF [Results-based Management and Accountability Framework] identifies possible methods for this evaluation. Proposers may suggest additional or alternative methods.", should be replaced by "The RFP also reads as follows:"
By order of the Tribunal,
Hélène Nadeau
Secretary
1 . R.S.C. 1985 (4th Supp.), c. 47 [CITT Act].
2 . 18 July 1994, C. Gaz. 1995.I.1323, online: Internal Trade Secretariat <http://www.intrasec.mb.ca/eng/it.htm> [AIT].
3 . S.O.R./93-602 [Regulations].
4 . See Re Complaint filed by Siemens Westinghouse Incorporated (19 March 2001), PR-2000-039 (CITT) [Siemens Westinghouse].
5 . GIR at 8, para. 30.