Before you spend half your paycheck at the trendy new restaurant downtown, chances are you first solicit opinions from friends who have already dined there.

Likewise with computer software selections…  talking with the vendor’s installed customers will help you separate "too good to be true" sales promises from the real world experience of actual users.

But whom do you ask… and what should you ask them? 

The following tips will help you get the most useful and candid information from vendor references.

Asking for References

It’s a good idea to ask vendors to provide you with references early in the selection process, such as the Request for Proposal (RFP) phase. One of the preloaded instructions to vendors in the ON-LINE CONSULTANT automated RFP software suggests:

"The references section should list ALL customers. Provide name, address, contact person, phone, software, hardware and date installed."

This request prevents vendors from providing you with only the hand-picked names of their most satisfied users.

In some cases you may find it more relevant to narrow down the customer list to sales within a recent time frame or in your region. Similarly, you can request the names of customers that are close in size and function to your organization. Your main goal is to get a well rounded sample of users--not just the vendors’ plum sites.

Keep in mind that talking to users who are dissatisfied with the vendor does not mean the vendor is at fault or the system would not be a good fit for you. System implementations often fail because the purchasing organization does not commit enough resources or key managers do not buy into the conversion--factors over which the vendor may have little control. If you talk to enough customers, however, you should get a good sense of overall satisfaction.

Each reference site includes a variety of people whose opinions of the software could be informative and helpful. Ideally, you should arrange for your selection committee members to speak directly with their counterparts from each of the customer sites so they can ask questions related to their area.

Speaking with several people from each organization also minimizes the impact of users whose opinions are overly critical or too generous, the sort of people who would be unhappy or delighted with whatever system they were using. With each member of your committee collecting a piece of the puzzle, you’ll be able to form a "big picture" view of the site’s overall satisfaction.

Reference checks can be performed at different points in the selection process. You can wait until you have already made a decision and use the reference check as a confirmation of your choice, or you can make the reference check part of your decision matrix.

If you use reference checks to confirm your choice, be sure to do your due diligence first. Perform a structured RFP process and invite vendors in for demos. Your references can be used to validate that vendor promises have panned out with actual customers.

If your reference calls will be used as part of you decision matrix, you should determine how much weight to assign to customer satisfaction. The weight for references usually ranges from five to 25 points, with other points allotted to RFP responses, vendor demos, etc. Just as with RFP responses, you’ll need to create a scoring method that will allow you to assign points to each response so you can derive a numeric score for each vendor.

It is best to make reference calls when you are down to two or three finalist vendors to avoid wasting other people’s time, not to mention your own.

References should be contacted by phone and asked to schedule an appointment for the conversation. Do not assume they will be available on short notice to discuss the vendors with you, though you should be prepared to ask questions if you have caught them at a good time.

When scheduling appointments, provide a time estimate for the conversations so the references can block out time on their calendars. In most cases, you should allow about 20 minutes for each call.

Follow up with e-mail confirmations so the references can contact you if they need to reschedule. You may elect to attach a list of the questions you will be asking so the references have time to think about their responses.

After the call, be sure to send thank-you notes (e-mail is okay), asking if you can recontact the references later should any new questions arise.

The hardest part of the reference process is formulating good questions. As a general rule, questions should be asked in an open ended manner to get more detailed information. Be sure to ask for both positive and negative experiences so the reference has "permission" to speak openly about all aspects of the system.

Your reference questions should cover the following topics:

Why they chose the software

System performance vs. expectations

Quality of training

Performance of implementation team

Ability of vendor to meet schedules and deadlines

Attitude of vendor staff (friendly, adversarial, etc.)

Problems during implementation… and how they were resolved

How bugs are handled

How new releases / upgrades are handled

Unexpected surprises (good and bad)

Challenge of finding and retaining IT talent to support the system

Major benefits of the system

Major limitations of the system

Vendor responsiveness to support and maintenance problems

Hidden costs

Customization issues

Members from your selection team will probably ask a narrower set of questions related to specific issues about which they are concerned.

If you find a user who seems dissatisfied with the vendor, try to find out what the specific problems were. How did the vendor handle the difficulties? Did they work as a team or leave the customer stranded? Did the vendor try to assign blame to others rather than take some of the responsibility themselves?

As a final question, it’s a good idea to ask your reference contact the following: "If you had it to do all over again, would you still choose the same company?"

In many cases a customer who has complained throughout the call would still choose the current vendor, either because they realize that no system works perfectly or they want to feel psychologically comfortable with their choice. Be sure to follow up with a "why?"… then remain silent to allow the customer to explain their response.

After you and your selection team have concluded your reference checks, assign someone to tally the scores and present the results in a report to your selection committee. During this meeting, members of the selection team should share anecdotes and important insights gleaned from their reference calls. These comments can be typed up and included in your report.

Like a good reporter, your ability to gain valuable information on a reference check will depend largely on the quality of your questions. Finding good references, asking probing questions and designing a uniform, structured scoring method will yield important data to help you make your final decision.

Copyright 2003 On-Line Consultant Software