How a Bakeoff Can Save you Money and Help You Make a Better Buying Decision
A large health insurer saved $300,000 unexpectedly. How did they do it? They conducted a live bakeoff.
The company was planning to purchase a new monitoring tool and was considering a handful of vendors. They had planned to evaluate each vendor’s tool in their live network one at a time to see how it would perform with real traffic. Ixia was brought in to mirror the virtual application traffic and deliver it to their monitoring tools. During conversations, we heard about the bakeoff and proposed using virtual taps and a network packet broker to provide concurrent copies of their live traffic to each vendor’s APM tool. This changed their serial evaluation to a concurrent test using the same live data so they could compare apples to apples. Ultimately, the company shaved 11 months off their proof-of-concept, which translated into $300,000 in labor costs. That is the power of a live bakeoff.
Before I discuss the value of a bakeoff, it is important to understand all the other components of a buying process because you cannot hold a bakeoff with tens or hundreds of products and solutions. You need a way to narrow it down to a handful of vendors. Your buying process (especially for an expensive solution) will likely be a system of techniques and evaluations. The buying process will vary depending on the company and the product being purchased, but will likely include vendor data sheets, peer reviews, third party tests, analysts, and possibly, a bakeoff. Let’s look at each one.
How to use vendor data sheets
Data sheets are some of the most heavily downloaded assets on a vendor website. They usually describe a product’s characteristics, capabilities, functions, and measurements – otherwise known as speeds-and-feeds. One thing to remember is that the speeds-and-feeds will likely include maximum values based on highly-controlled lab testing that will demonstrate the highest performance for the system under test. Making a complicated buying decision based solely on vendor data sheets can be dangerous. Caveat emptor applies. Instead, use data sheets as a guide to find out what devices will work and will not work based on your environment or your purchase criteria.
How to use your peers’ experience
I love to see what my peers think about a product or service. There are always complaints and praises. One of my favorite sites for reading what other people think about vendor technology is Spiceworks. IT Pros will ask questions about a product and others who have used it will respond with candor. Peers provide insights that go beyond a data sheet. For instance, if a product’s interface is difficult to use, you will find out. If a product had problems after 6 months of operation, you will read about it. Interoperability issues? Someone will mention it. But just like data sheets, peer ratings and recommendations should be used only as a component in the overall buying process. Also, be careful of peer ratings with fat tails and a neutral rating. This means everyone either really hates it or really loves it, but its overall rating is average.
How to use third party test lab reports
Third party test labs will put a group of products you may be evaluating through a standardized testing methodology. Then, they will publish the results and the methodology they used. The tests and reports are looking to stack similar devices from different vendors against each other to see how they compare. These reports provide a lot of information you will not likely get directly from the vendor. As with data sheets and peer reviews, test lab reports may be slightly skewed towards the testing of a feature or capability under a specific condition. This is especially true if it is a sponsored report. There is nothing nefarious about a third party test lab doing this, but you may not be getting the whole story. Try to keep the tests in context of the setup and results documented.
How to use analysts and their reports
Analysts can be the great equalizer to a one-way conversation with vendor sales reps. They can help narrow down your buying choices by expanding on the indirect aspects of the purchase. They can point out the negatives. Sometimes they will function as a customer advocate. Analysts will know how the sausage is made, what is going on with a vendor, what is truthful, what is not truthful, what is the uproar, the politics, who is getting acquired, who wants to get acquired, who is having a big shakeup in their organization because they are not hitting their numbers, etc. Analysts can help you get to the other side of it.
What are the different types of bakeoffs?
A bakeoff can be one of the most important steps in your buying process. Once you have narrowed the field down to a few vendors, identifying the one best suited for your environment can only be gleaned by holding a bakeoff. There are two types of bakeoffs.
- A lab bakeoff is the process of using synthetic data or replays of live data with devices under evaluation. This can be part of a functional, performance, or security test. This is a great way to understand how devices handle cross-cuts of data found on a data sheet or how they handle traffic not found on production networks (like security strikes). For instance, if you were selecting a firewall, you might measure the percentage of attacks blocked at top speed or how well it protects against a flood of DDoS traffic.
- A live bakeoff is the process of using live data with devices under evaluation. Using a visibility network, copies of network traffic are forwarded to a network packet broker which duplicates the packets to each device. This is a great way to understand how each of the devices perform in your environment under real conditions. Live bakeoffs can be run for months as tweaks are made and data are analyzed.
How to use a bakeoff to your advantage
The key to making a great buying decision is to learn the actual capabilities of the device under your company’s own network conditions – not in the vendor’s lab. Exercising various stress points with real applications will show how well each system under test performs for these metrics. For example, if you are testing an IPsec gateway, then the connection setup rate is as important as the encrypted data throughput. If the connection rate delta between two devices tested is 10x, the worst performing device will ultimately be a lot more expensive to own. A bakeoff will help you uncover these details.
Bakeoffs provide 3 distinct benefits:
- Gain greater negotiating power. With a bakeoff, you get into a competitive situation. When your purchase is competitive, you may be able to lower your purchase price. You can use the measured performance observations from the bakeoff as a negotiation tool with the vendor. For example, if the measured performance in a given category is 30% less than what is advertised in the vendor data sheet, you now have additional leverage.
- Take it for a test drive. While paying the best price is important, you also need to test drive it. Is it hard to manage? While the CAPEX looks great, because it is difficult to operate and manage, maybe the OPEX will be really high over the three years it stays on your books.
- Are you buying enough? Too much? Testing with real-world traffic promotes rightsizing because it allows engineers to build in enough of a performance cushion to handle peak traffic without creating waste by overbuilding that cushion.
Ixia provides a complete suite of pre-deployment test products like IxLoad, IxNetwork, and BreakingPoint to run a lab bakeoff on your terms. We also have the visibility architecture, IxVision, to create and run live bakeoffs.