Bill Alderson, Technology Consulting Officer, NetQoS, Inc.
Catalyst symptom: Outlook 2003 issuing client/server access warnings.
First question: Is the network stable?
First analysis: Monitor from several end points on the network to determine if connectivity is intermittent by monitoring at a 10 second interval 24/7.
Result: Yes, outages occur.
Second question: Where are packets lost?
Second analysis: Triangulate packet loss location by examining the monitor logs and testing the suspect circuits with Spirent tools.
Result: Spirent's Smartbits platform and Avalanche software found that their POS (packet over sonet) links had asymmetrical throughput. Instead of 155 mbps in both directions, one direction was 30 kbps and the other near 155 mbps.
Third question: Why are we getting only a fraction of the bandwidth we are paying for?
Third analysis: Determine the components responsible for the link and examining the configuration details.
Result: Cisco 7513 interfacing with a 7206. The direction of the low throughput was toward the 7206s from the 7513s. Verified that 7513 to 7513 had full bandwidth symmetrically in both directions.
Thesis: A directional issue between a 7513 and a 7206 POS connection.
Result: It was indeed a problem with the routers as we suspected. It took us 3 days to get the Cisco TAC to come to the same conclusion. We all had a good time laughing as these words came over the speakerphone, “looks like a directional problem between the 7513 and the 7206.” It took 3 days to get them to believe our analysis steps and results.
Solution: You guessed it, a new release of code finally did the trick.
Cost to the client: Two latent years of TEN packet over sonet links running at less than 30 mbps instead of 155 mbps supporting 9,000 users.
Question: Which cost more, the lost productive time of 9000 users or the payment for a circuit difference between 155 and 30 mbps?
Challenge: How can such an analysis be quantified to show the ROI for the $12,000 PMG charged for this specific analysis?
If you were the manager of the department, how much budget for tools and training to diagnose such problems would you consider to be a good ratio to spend to diagnose the problem in house?
Do you think it is cost-effective to have an expert with a wide variety of experience come in to audit and help troubleshoot problems quarterly?
NetQoS - Network Performance Management products and services for the world's largest networks. © 2001-2008 NetQoS, Inc. All rights reserved.