News Detail

News Archive >> WLAN stress test uncovers 802.11 performance problems

Details

WLAN stress test uncovers 802.11 performance problems
The tests confirm two troubling issues for high-density nets
John Cox (Network World) 13/02/2008 11:32:52

http://www.computerworld.com.au/index.php?id=1526372596&eid=-246


A rare large-scale, wireless LAN stress test using three vendors' equipment found many WLANs could run into a performance ceiling as they grow in size and traffic. The study's authors say the problem lies not with the vendors' gear but with the design of the 802.11 protocol.
The tests confirm two troubling issues for high-density nets, according to Novarum, a consulting firm that developed and ran the tests. First, co-channel radio interference among the access points clobbers the aggregate throughput of the WLAN. Second, the conventional thin access point/controller architecture doesn't scale well as the number of access points increase in a given area.
According to Novarum, performance degrades because of radio interference in dense deployments, and because the conventional model for enterprise wireless LANs -- thin access points linked with a controller -- scales poorly.
These problems are related to the way the 802.11 media access control (MAC) layer is designed, how it handles acknowledgements and retries, and how these become problematic under high and sustained traffic loads. In many WLAN products, and the WLANs built with them, this protocol inefficiency hasn't been an issue, because there have been relatively few wireless clients and their traffic has consisted of relatively light, and bursty, data traffic.
High-throughput WLANs based on 802.11n, especially running in the 5GHz band, will partly mitigate these effects, according to the study's authors, but not entirely. Because 11n offers a larger 'data pipe,' it takes more traffic to reach network overload in a dense WLAN. But because of 11n's higher throughput, enterprises are looking to do more with it, such as voice and streaming video, which can push the net toward overload.
"If you don't deal with co-channel interference and [the need for] access point coordination, it will put more pressure on the protocols supporting [wireless] voice and video," says Phil Belanger, co-founder of Novarum.
Designing a real-world test
Novarum's tests, conducted in the fall of 2007 in a vacant second-floor office in Sunnyvale, Calif., are unusual because of the number of access points and clients actually deployed. As the report points out, WLAN tests typically involve a single access point and about a dozen clients deployed in a lab-like environment.
In this case, Novarum used 72 wireless laptops and up to 54 wireless VoIP handsets, linking to a typical office wireless LAN first with 15 and then with 10 access points. The net was recreated each time with access points from Aruba Networks, Cisco, and Meru Networks.
Novarum designed and ran the tests. But the project was paid for by Meru (the office space had been rented by Meru for employees who had not yet moved in).
Seven tests were run, in most cases using first 15 and then 10 access points. The access points had 802.11abg radios, but Novarum ran the test only for 11g in the 2.4GHz band. One test was data only, with the 72 laptops; others tested only voice traffic, with 24, 48 and 72 simulated VoIP conversations. Two tests mixed voice and data, and one tested the VoIP handsets to find out how many simultaneous calls the WLAN could support.
In e-mails to Network World after the original version of this story was posted online, Aruba's head of strategic marketing, Michael Tennefoss, objected to what he called a "biased report" based on "bad science," and a "Meru puff piece." We invited Aruba to post its objections in the reader comments mechanism at the end of the story online, but Tennefoss declined, saying readers would interpret the post as the "sour grapes" of the "loser."
There were a number of technical objections Tennefoss makes, but one stood out. Tennefoss said the antennas on the Aruba access points were in the "closed" position (that is, folded on its hinge against the case) instead of in the "open" position as instructed in the Aruba users's guide. "They might as well have removed the antennas altogether," he writes.
He knows this was done because the Novarum report explicitly mentions it, in Appendix C. That entry says the tests were initially run with the antennas correctly positioned but the data throughput results were "very disappointing." The results improved dramatically when the antennas were folded down, so the tests were run with the antennas in this position.
Tennefoss' says that this behavior should have been a clue to the fact that something was not configured properly or the access points were wrongly deployed.
In an e-mail response, Novarum's Belanger says the stress test was not intended as a product review of the vendors' equipment, though they took a similar starting point: "standard [vendor] product tested with the latest released software and standard tools. Whenever we deviated from that it was explicitly called out [in the report]," he writes.
He defends testing the Aruba access points with their antennas folded because it resulted in much better performance for the Aruba WLAN.
In our own large-scale WLAN test in 2006, with 25 access points, Aruba won the Network World Clear Choice Award for its performance. Meru was the only other vendor, out of 19 invited, willing to participate, but various problems including beta software code prevented that vendor from completing the test. Full test results are online, including deficiency in the 802.11 protocol that could lower throughput.
Thresholds appear
"There seems to be a threshold, and beyond that the [WLAN] system doesn't behave well," says Novarum's Belanger. "The bigger the load, the more errors [are introduced], leading to more retransmissions, and this [in turn] adds to the load. And that adds to still more errors."
The brands of WLAN gear handle this spiraling effect differently, depending on the kind of network architecture they use. Aruba and Cisco use what Belanger calls a microcell architecture: a stripped-down access point linked with a centralized controller. Adjacent microcells run on separate channels, with some coverage overlap to provide seamless coverage and roaming for mobile clients. This model is used by most of today's WLAN vendors.
Meru by contrast can set all the access points to run on a single channel, and exercises much more control over its "on-air" behavior. The Meru controller can see the transmit queues on each access point for each associated client, and can allocate how much time the access point devotes to each, according to Belanger. Instead of trying on their own to get through the transmit "door," and jamming into it, the Meru access points patiently wait for their turn, so throughput is steady, predictable and optimal. That's the theory.
One other vendor, Extricom, takes a somewhat similar approach, though there are significant differences from Meru. Extricom packs four radios into a single device but runs the entire 802.11 MAC in its controller. The access point is nothing more than the radios and antennas; it even lacks a CPU. Extricom claims to eliminate co-channel interference and to optimally manage client wireless connectivity on a per-packet basis.
Test results
According to Belanger, in the data-only test, with 72 clients and 15 access points, Cisco and Aruba delivered less than 50Mbps of total system throughput. In other words, networkwide, each client was getting on average less than 1Mbps throughput. One would expect something like 20Mbps throughput per access point, or 300Mbps aggregate for all 15, Belanger says.
But when the number of access points was decreased to 10, throughput jumped. In Aruba's case, it jumped nearly 40%, from 47Mbps to 64Mbps. "More [access points] allow more simultaneous transmissions, which caused more interference and lowered performance in these systems," the report says.
In Meru's case, the office was covered by five access points, all running on the same channel. Two more sets of five access points were added in the same locations, each set with a different one of the two remaining 2.4GHz channels. All ran at their maximum power setting. Meru delivered 100Mbps of system throughput, or more than two times that of Cisco and Aruba. Interestingly, Meru's aggregate dropped to 60Mbps when Novarum reduced the access points to 10 devices.
"It appears that co-channel interference from adjacent APs has a significant impact on the performance of the micro-cellular systems when they are loaded," the Novarum report concludes. Belanger notes the interference range of an 802.11 radio far exceeds its effective communications range. "Under heavy, constant loads, this interference becomes a factor," he says.
According to Belanger, at some points in these highly loaded networks, 30 per cent to 40 per cent of the packets in the air were transmissions by the Aruba and Cisco access points. Meru had far fewer retransmissions.
Voice calls run into limits, too
In general, the Meru architecture also offered better voice support, being able to handle more simultaneous calls (Novarum didn't encounter an upper limit in the tests), and deliver toll quality, according to Belanger. With 10 Cisco access points, Novarum found the Cisco WLAN able to handle about 24 VoIP calls. With 48 or more simulated calls, Cisco was unable to deliver toll quality. When real handsets were tested, the Cisco infrastructure seemed to top out with 26 or 28 simultaneous calls.
In the report's appendix, Belanger writes, "I was surprised to see how easily we drive these systems to unstable behavior" with lots of access points and high, sustained loads, a problem he sees related to the 802.11 MAC protocol he championed. "There is nothing wrong with the Aruba and Cisco systems," he writes. "They simply chose not to address this [issue]."
Work by two IEEE 802.11 task groups, 11k and 11t, will address it, at least in part, by making it possible to exercise more control over access point radios and to some degree over client radios. That work is still in progress.
The question of bias
How much reliance one puts on the Novarum stress test depends on two issues: whether you agree with the testers' assumptions, and your assessment of Meru's influence on the test and the independence of the testers.
For example, in the "Conclusions" section of the test report, Belanger writes "The only unusual thing about our testing is the constant load during the data tests. That type of load is similar to 72 people on the same network downloading a movie from [the] iTunes store at the same time." Belanger himself admits this is not typical behavior for most enterprise networks but goes on to write "we would expect any of these systems to be able [to] handle it gracefully."
Was the test fair to Cisco and Aruba? Meru engineers configured and tuned the Meru equipment, a benefit not accorded to Aruba and Cisco.
Belanger acknowledges the appearance of bias but says the tests really were designed to test the optimal configurations for Aruba and Cisco, and were mainly concerned not with rating the three products but with the behavior of the 802.11 protocol under load, and how different architectures dealt with that behavior.
"[Meru executives] wanted it to be about 'our product is better,'" he says. "But we were interested in the larger issue of the different architectures."
In an appendix to the complete test report, called "Full Disclosure," Belanger says his personal bias has been "in the micro-cellular, Cisco and Aruba camp." That's in part because he acknowledges he was a major proponent of this approach during development of the 802.11 standard. While at Aironet (later bought by Cisco), Belanger says he led the company's work on developing micro-cellular software for early wireless products, and much of that code is still part of the Cisco Compatible Extensions software.

Date

Thursday 14 Feb 2008 - Saturday 14 Feb 2009