You need to be careful to avoid the trap of looking at an IBR (Integrated Baseline Review) as something that is graded like a term paper or assigning a “Pass/Fail” grade to the process. The IBR is meant to be a collaborative effort that consolidates the understanding of a baseline, assumptions and risks. Grading that conversation at the end will skew the process to doing just what needs to be done to pass, which is usually not the most open conversation.
Having said all that, scoring the individual interviews can add value to your IBR process. The goal of the scoring isn’t to ultimately assign a pass/fail grade, but instead to capture information that will help you identify areas of concern and/or trends. This is particularly important if you are doing an IBR on a large program that requires multiple Control Account Manager (CAM) interviews to be held at the same time. If you have a consistent scoring process in place going into the interviews then you be in a better position to compare the results of the two interviews even though the audiences were different.
The scoring should be captured for the 5 major areas on which you are reviewing the CAM during the interview: Scope, Schedule, Budget, Resources and Process. Scope being based on the CAM’s understanding of the scope associated with the control account and the assumption used in the planning. Schedule focused on quality and how realistic the schedule is. Budget focused on the dollars planned for the control account and assumptions used to justify that level. Resources focused on the availability of resources needed to do the work and processes focused on how well the CAM knows and uses the earned value processes.
Weight each area based on a simple scale that allows reviewers to quickly identify with a level. Typically we use a 5-point scale where 1 is a high level of confidence in the plan and 5 is a very low level of confidence in an area. It will help the reviews if you can give them some parameters prior to the review about what would typically be graded a 1 versus a 5.
For example, if the CAM can not only answer all the questions about the Earned Value (EV) process but also talks to how they use the process in their day to day management that would be a 1. If they struggle to spell EV, that would be a 5. Don’t get too caught up in the exact definition of each level, what you’re looking for are trends more than exact scores. You’ll ultimately average the scores across reviewers, which will help towards normalizing the results.
All of the designated interviewers should fill out the forms based on their impression of the interview. Having each person fill out the form both expedites the process by avoiding lengthy discussions about what the appropriate score should be as well as avoid the answers always being skewed to the loudest voice in the room. At the end of the IBR, or the end of each day if it’s a multiple day review, the scores can be consolidated for each CAM interview and the average score by CAM or area can be analyzed.
The screen shot below is an example of a typical scoring sheet that you can use to capture the inputs. You can use this by control account or by CAM interview if the CAM is responsible for more than one control account. It just depends on how you want to structure things.
For each score of a 5, a low level of confidence, the review should document a Corrective Action Report. If the interview was that bad there should be some specific steps that should be taken to improve the confidence. We usually expand the top table portion to include the control accounts for which a CAM is responsible and complete one of these forms per CAM interview. That is the most intuitive for the reviewers.
In addition to helping you identify trends in the program, for example if all the CAM’s score poorly on the process piece, you can use these scores as inputs into your Schedule Risk Assessment (SRA). Take the average score for the cost and schedule areas and use that as input for the confidence levels during your SRA or Joint Confidence Level (JCL) reviews.
You can download a soft copy of the scoring template here.