Expert’s Guide to Evaluating Software: Chapter 8

Expert’s Guide to Evaluating Software: Chapter 8

20 Questions You Should Ask Vendors

RFIs, RFPs, and pilots. Bake-offs, ROI/TCO calculations, demos, and feature comparison checklists. These can all be effective software evaluation tools, but not every company uses them the same way.

When making a strategic software investment, the evaluation process often varies but the objective is the same: find the best solution for my company.

We encourage customers to use various techniques to both de-risk their technology investments, and expose the weaknesses of vendors who might appear strong on paper but can’t match up on a real-world use case.

To establish to a common ground, let’s start by shifting the buyer’s mindset.

Approach software evaluations like you are interviewing a potential employee. Look as their past. Their pedigree. Get references, ideally blind. And get to know them to assess the impact they can make in the future, how their goals align to your company’s, and culture fit (open API, fast, scalable, flexible, etc).

Don’t make a bad hire, as they are more costly than doing sufficient due diligence. To help; below is a list of 20 general questions for software vendors. They are market and application agnostic.

1. How many versions of your software are you supporting?
Best answer is one, as this will give you the fastest pace of innovation.

2. What countries do you have users in?
The more the better as this speaks to scale and global performance.

3. What is your upgrade or release process? How often do you do it?
Flush out any hidden costs or potential disruption as well as pace of innovation.

4. What are the ongoing costs in addition to licenses?
Lower costs are better but also assess cost predictability.

5. How many customers do you have, and what is their collective volume of data/content?
More the better to demonstrate scale and execution.

6. What is the background of your leadership team?
Leadership experience in software matters. The more patterns they’ve seen, the better.

7. How many different software delivery approaches do you have?
Best answer is one. Red flag if a vendor has on-premise, private cloud, and multi-tenant cloud. This show a lack of focus.

8. Where is the development team based?
Communication is integral to success software companies. Needs come from customers go to product designers, which go to architects, which then get coded by developers, which are then tested by QA teams. This relay can lose fidelity if communication is not crystal clear.

9. Is your company profitable or cash flow positive?
A software vendor in good financial standing has the means to invest more than those that are not.

10. What is the ownership profile of your company?
Are they public, VC-backed, or private equity owned? Public is safest as nothing is hidden. Private equity is riskiest because a banker is captaining a software investment ship.

11. How quickly is your revenue growing year over year?
While customer and employee growth is also good to understand, the numbers can be misleading as other variables can be at play. Revenue growth is the purest motivator for ongoing investment for a software company.

12. Does the software offer an open, public API at no additional cost?
Systems are meant to talk to each other. In 2017, an open API is a must. If a vendor charges for the API then what else are they going to nickle and dime you on?

13. How is the software modified/customized?
Is it by configuration, code, or both? If both, find out what the configuration is used for and what the code is used for.

14. Is the application built on an application platform or is it a point solution?
Many vendors will call an application a platform. The quick test is to ask how many applications are on the same platform...and don’t let a vendor call a module or add-on an application.

15. What customer use metrics does the vendor track?
Performance should definitely be one.

16. Is the licensing perpetual, named, or concurrent?
Is it based on data volume, logins per time period, or something else? Perpetual is usually for on-premise. Some vendors offer concurrent licenses, and watch out for those. If you exceed your concurrent licenses due to an event, you may be looking at large “roaming” or overage charges…..or worse, your users can’t get in.

17. Where is the software hosted?
The best are AWS, Microsoft Azure, Google Cloud, NTT data centers, etc.

18. Does the vendor provide transparency into the status of the service?
How will you know if the system up, down or sideways? Pure cloud vendors will often provide a public site that details the status of the service and will report on any disruptions to the service. Here is an example:

19. Does the vendor have a robust partner ecosystem?
System integrators and other 3rd parties can play a significant role in the tuning of a SaaS solution. Don’t be trapped and sole-sourced by your vendor. Make sure that there is an ecosystem of partners who can help you to configure and fold the service into your business environment.

20. What is the vendor’s customer renewal rate?
In the world of SaaS, customer renewal rates are a proxy to customer satisfaction. If a customer is unhappy with a service, then they won’t renew their subscription with a vendor. Also, if a vendor is lucky enough to have “negative churn,” then they are expanding their footprint within accounts, as opposed to shrinking or not renewing. Negative churn is the best indicator for both customer satisfaction and adoption.

Armed with these 20 questions, any software buyer can select the best software vendor for their company's needs.


Expert's Guide to Evaluating Software Series:

Chapter 1: What Really Matters When Making Software Investments
Chapter 2: Defining Software Models
Chapter 3: Security & Scalability
Chapter 4: Total Cost of Ownership
Chapter 5: Innovation
Chapter 6: Flexibility
Chapter 7: Conclusion
Chapter 8: 20 Questions You Should Ask Vendors


Frank Defesche began his software career at Trilogy Software in Austin, TX, an on-premise software company. In the summer of 2000 he joined as one of their first consultants. In a world dominated by on-premise and home grown software, he was faced with the challenge of translating traditional software processes to an emerging cloud paradigm. He was part of the cloud’s first chapter and has lived it ever since. He currently serves as the SVP and General Manager of Veeva Systems and is responsible for expanding Veeva’s solutions to industries beyond Veeva’s life science beginnings.