ProSafe Network Management System Data Sheet NMS200
The NETGEAR Network Management System NMS200 allows for smarter management of NETGEAR infrastructures. Powerful, scalable, and simple to use, this management platform is installed and ready to use in a matter of minutes. NMS200 offers a single console for extensive visibility, granular control, and seamless automation across the entire network. By normalizing how the infrastructure is discovered, congured and monitored, administrators can save time, reduce errors, and increase operational efciency.

Discovery and Mapping

NMS200 automatically discovers all NETGEAR-compatible equipment, interfaces and physical and logical connectivity, using various protocols and centralized credentials. NMS200 periodically re-syncs with the environment to ensure accurate network integrity. IT administrators can maintain a complete inventory of NETGEAR-managed infrastructure and easily view the network environment in the customizable topology viewer. The NETGEAR Network Management System displays the network logically and geographically enabling IT teams to see conditions as they occur and gain additional network knowledge by drilling down in detail with a mouse click to subcomponents and logical connections.

Configuration

NMS200 conguration management capabilities let administrators automatically maintain, back-up, restore, and compare network device congurations, eliminating the inefciency and errors associated with manual conguration. With its easy-to-use graphical user interface, administrators can congure devices, save, and update conguration data with direct access. The NETGEAR Network Management System also centralizes rmware/OS management: rmware versions can be deployed to individual devices or groups of devices, based on a schedule.

Monitoring

The NMS200s built-in monitors accurately collect and analyze real-time health-affecting conditions and performance metrics for NETGEAR managed equipment, indicating current and historical performance. The NMS200 provides graphical dashboards for key device performance indicators as well as service performance metrics, to consistently ensure the network is meeting expectations. Based on pre-dened and user-dened criteria, the management platform escalates appropriate issues as alarms via multiple notication methods the alarm viewer, via email or pager, to 3rd-party systems, etc. System-initiated operations such as script-execution, conguration backups and/or restorations can be automatically triggered, while NMS200 displays dependable and lterable records of all events and alarms that have occurred in the environment, for troubleshooting and remediation purposes.
The NMS200 platform is downloadable at www.netgear.com/nms200. This is the fully functional version, with a 5-device test capability and no expiration date. License packs are available from 15-device up to 500-device, and can be easily mixed and matched to meet todays and tomorrows growing network needs.

TECHNICAL S U P P O R T*

24 / 7
1-888-NETGEAR (638-4327) Email: info@NETGEAR.com
ProSafe Network Management System

NMS200

TECHNICAL SPECIFICATIONS OPERATING ENVIRONMENT OS Support Windows Server 2003 32/64-bit Windows Server 2008 32/64-bit Windows XP 32-bit Windows Vista 32/64-bit Windows 7 32/64-bit
GUI-based Installation Congurable Installation Location Management Interface Support DISCOVERY Automated Device Discovery Automated Link Discovery Discovery Protocol support Discovery Scheduling Device Resynchronization Device Resynchronization Scheduling SNMP MIB Browser MONITORING Topology Mapping Event Monitoring Alarm Escalation Alarm Propagation Alarm/Event Actions Device Performance Key Metrics Active Performance Monitoring Canned Reports CONFIGURATION Conguration File Backup Conguration File Restoration Device Firmware Update Firmware Pre-load Conguration File Comparison Device conguration SUPPORTED DEVICES Fully Managed Switches
Automated (single server deployment - Windows) Default install directory/path can be changed by user during installation process SNMP v1/v2/v3 CLI: Telnet/SSH Automated device discovery -- includes top-level, subcomponents and interfaces/ ports as applicable Ethernet link discovery with LLDP LLDP Support Ability to schedule discovery tasks to be executed at specied time/date(s) in the future System resynchronization with device inventory Ability to schedule device resynchronization to be executed at a specied time/ dates(s) in the future Includes SNMP MIB browser for users to quickly view attributes of all MIBs in device Geographic and logical topology views, including ltering SNMP trap reception with dened trap attribution, severity and descriptions Alarm generation based on pre-dened event denitions Alarm propagation to alarm viewer and topology views Event/alarm-initiated pre- or user-dened actions: notication, conguration operations, etc. Real-time key performance metric collection and display for interfaces/ports of selected devices Device & interface monitoring, historical data persistence, thresholding & graphing Pre-dened reports for inventory, availability, port status, etc. Device conguration le backup for applicable devices; 1:1 and 1:Many, immediately or scheduled Device conguration le restoration for applicable devices; 1:1, immediately or scheduled Automated deployment of selected version of rmware, immediately or scheduled Firmware versions may be pre-loaded into the application Color-coded, line-adjusted text comparison of two selected devices or stored les Direct access/cut-through (HTTP/Telnet/SSH) FSM726-300 GSM7224-200 GSM7248-200 GSM7228PS GSM7252PS GSM7328FS GSM7328S-200 GSM7352S-200

Smart Switches FS726T * FS726TP * FS728TP GS108T-200 GS110TP GS716T-200 GS724T-300 GS748Tv3 * GS724TP GS748TP GS724TS GS748TS GS724TPS GS748TPS * Limitations apply, please refer to the Release Note for details on the software download page - www.netgear.com/nms200
SUPPORT ENTITLEMENT Warranty Technical Support NETGEAR 90-day Warranty Basic technical support within 90 days from the date of NMS200 License(s) purchase Basic technical support within 90 days from the date of supported NETGEAR devices purchase Basic and Advanced technical support when NETGEAR Managed devices are covered by the ProSupport OnCall 24x7 service contract NETGEAR ProSafe Network Management System NMS200 v2.1 SKU: NMS215-10000S SKU: NMS230-10000S SKU: NMS250-10000S SKU: NMS2100-10000S SKU: NMS2500-10000S
ORDERING INFORMATION Current NMS200 version 15-device license pack NMS215 30-device license pack NMS230 50-device license pack NMS250 100-device license pack NMS2100 500-device license pack NMS2500 License pack utilization Licensing information
Device license packs are electronic keys Each electronic key needs to be redeemed at www.doradosoftware.com/netgear Device licenses can be mixed and matched 5-device built-in capability is intended for testing only The rst device license will reset the NMS200 device counter For instance, the rst NMS250 license will reset NMS200 device counter to 50 managed devices additional device licenses are incremental

Upgrades

Free NMS200 v2.x minor upgrades included
350 E. Plumeria Drive San Jose, CA 95134-1911 USA 1-888-NETGEAR (638-4327) E-mail: info@NETGEAR.com www.NETGEAR.com
2010 NETGEAR, Inc. NETGEAR, the NETGEAR Logo and ProSafe are trademarks and/or registered trademarks of NETGEAR, Inc. and/or subsidiaries in the United States and/or other countries. Mac and the Mac logo are trademarks of Apple Inc., registered in the U.S. and other countries. Other brand names mentioned herein are for identification purposes only and may be trademarks of their respective holder(s). Information is subject to change without notice. All rights reserved. *Free basic installation support provided for 90 days from date of purchase. D-NMS200-1

packets of a ow need to arrive before the deadline. 2. Deadlines for ows can vary signicantly. For example, online services like Bing & Google include ows with a continuum of deadlines (including some that do not have a deadline). Further, datacenters host multiple services with diverse trac patterns. This, combined with the previous challenge, rules out traditional scheduling solutions, such as simple prioritization of ows based on their length and deadlines (EDF scheduling [9]). 3. Most ows are very short (<50KB), and, at the same time, RTTs minimal (300sec). Consequently, reaction time-scales are short, and centralized, or heavy weight mechanisms to reserve bandwidth for ows are impractical. In this paper, we present D3 , a Deadline-Driven Delivery control protocol, that addresses the aforementioned challenges. Inspired by proposals to manage network congestion through explicit rate control [6,10], D3 explores the feasibility of exploiting deadline information to control the rate at which endhosts introduce trac in the network. Specically, applications expose the ow deadline and size information at ow initiation time. Endhosts use this information to request rates from routers along the data path to the destination. Routers thus allocate sending rates to ows to greedily satisfy as many deadlines as possible. Despite the fact that ows get assigned dierent rates, instead of fair share, D3 does not require routers to maintain per-ow state. By capitalizing on the trusted nature of the datacenter environment, both the state regarding ow sending rates and rate policing are delegated to endhosts. Routers only maintain simple aggregate counters. Further, our design ensures that rate assignments behave like leases instead of reservations, and are thus unaected by host or router failures. To this eect, this paper makes three main contributions: We present the case for utilizing ow deadline information to apportion bandwidth in datacenters. We present the design, implementation and evaluation of D3 , a congestion control protocol that makes datacenter networks deadline aware. Results from our testbed deployment show that D3 can eectively double the peak load that a datacenter can support. We show that apart from being deadline-aware, D3 performs well as a congestion control protocol for datacenter networks in its own right. Even without any deadline information, D3 outperforms TCP in supporting the mix of short and long ows observed in datacenter networks. 2

Figure 3: An example of the partition aggregate model with the associated component deadlines in the parentheses. While we are convinced of the benets of tailoring datacenter network design to the soft real time nature of datacenter applications, we also realize that the design space for such a deadline-aware network is vast. There exists a large body of work for satisfying application demands in the Internet. While we discuss these proposals in Section 3, the D3 design presented here is heavily shaped by the peculiarities of the datacenter environment; its challenges (lots of very short ows, tiny RTTs, etc.) and luxuries (trusted environment, limited legacy concerns, etc.). We believe that D3 represents a good rst step towards a datacenter network stack that is optimized for application requirements and not a retrotted Internet design.
BACKGROUND: TODAYS DATACENTERS
otherIn this section, we provide a characterization of todays datacenters, highlighting specic features that inuence D3 design.
2.1 Datacenter applications
Partition-aggregate. Todays large-scale, user facing web applications achieve horizontal scalability by partitioning the task of responding to users amongst worker machines (possibly at multiple layers). This partitionaggregate structure is shown in Figure 3, and applies to many web applications like search [5], social networks [11], and recommendation systems [1]. Even data processing services like MapReduce [12] and Dryad [13] follow this model. Application deadlines. The interactive nature of web applications means that latency is key. Customer studies guide the time in which users need to be responded to [14], and after accounting for wide-area network and rendering delays, applications typically have an SLA of 200-300ms to complete their operation and ship out the

40 Mean Deadline (ms) 50

Figure 4: Deadlines met (%) based on ow completion times from a production datacenter. user reply [1]. These application SLAs lead to SLAs for workers at each layer of the partition aggregate hierarchy. The worker SLAs mean that network ows carrying queries and responses to and from the workers have deadlines (see Figure 3). Any network ow that does not complete by its deadline is not included in the response and typically hurts the response quality, not to mention the wasted network bandwidth. Ultimately, this aects operator revenue; for example, an added latency of 100ms costs Amazon 1% of sales [4]. This leads to Observation 1: Network ows generated by applications have deadlines and these deadlines are (implicitly) baked into all stages of the application operation. Deadline variability. Worker processing times can vary signicantly. For instance, with search, workers operate on dierent parts of the index, execute dierent search algorithms, return varying number of response items, etc. This translates to varying ow deadlines. Datacenter trac itself further contributes to deadline variability. We mentioned the short query and response ows between workers. Beyond this, datacenter trac comprises of time sensitive, short messages (50KB to 1MB) that update the control state at the workers and long background ows (5KB to 50MB) that move fresh data to the workers [5]. Of course, the precise trac pattern varies across datacenters. A Google datacenter running MapReduce jobs will have primarily long ows, some of which may carry deadlines. Hence, Observation 2: Todays datacenters have a diverse mix of ows with widely varying deadlines. Some ows, like background ows, may not even have deadlines. Missed deadlines. To quantify the prevelance of missed deadlines in todays datacenters, we use measurements of ow completion times in datacenters [5]. We were unable to obtain ow deadline information, and resort to capturing their high variability by assuming that deadlines are exponentially distributed. Figure 4 shows the percentage of ows that meet their deadlines with varying mean for the deadline distribution. We nd that even with lax deadlines (mean=40ms), more than 7% of ows miss their deadline. Our evaluation (section 6) shows that this results from network 3

3. DESIGN SPACE AND MOTIVATION
The past work for satisfying application deadlines in the Internet can be categorized in two broad classes. The rst class of solutions involve packet scheduling in the network based on deadlines. An example of such scheduling is Earliest Deadline First (EDF) [9] wherein routers prioritize packets based on their per-hop deadlines. EDF is an optimal scheduling discipline in that if a set of ow deadlines can be satised under any discipline, EDF can satisfy them too. The second solution category involves rate reservations. A deadline ow with size s and deadline d can be satised by reserving s rate r = d. Rate reservation mechanisms have been extensively studied. For example, ATM supported Constant Bit Rate (CBR) trac. In packet switched networks, eorts in both industry (IntServ, DiServ) and academia [17,18] have explored mechanisms to reserve bandwidth or at least, guarantee performance. Value of deadline awareness. Given this existing body of work, we attempt here through simple Monte Carlo simulations, to build some intuition regarding the (possible) benets of these approaches over fair sharing used in datacenters today. Consider a 1Gbps link carrying several ows with varying deadlines. Flow parameters (such as the size and the fraction of short and long ows) are chosen to be consistent with typical datacenters [5]. For the ows with deadlines, the deadlines are chosen to be exponentially distributed around 20ms (tight), 30ms (moderate) and 40ms (lax). To capture the lack of application value of ows that miss their deadline, we use application throughput or the number of ows that meet their deadline as the performance metric of interest. Using this simple simulation setup, we evaluate three ideal bandwidth allocation schemes: (i) Fair-share, where the link bandwidth is allocated evenly amongst all current ows and represents the best-case scenario 4
acenter networks, where network conditions change very fast (e.g., tiny RTTs, large bursts of short ows). Overall, these limitations motivate the need for a practical datacenter congestion control protocol that, on the one hand, ensures ows meet their deadlines, but, on the other, avoids packet scheduling and explicit reservations.

4. D3 DESIGN

The discussion in the two previous sections leads to the following goals for datacenter congestion control: 1. Maximize application throughput: The protocol should strive to maximize the number of ows that satisfy their deadlines and hence, contribute to application throughput. 2. Burst tolerance: Application workows often lead to ow bursts, and the network should be able to accommodate these. 3. High utilization: For ows without deadlines, the protocol should maximize network throughput and achieve high utilization. D3 is designed to achieve these goals. Beyond these explicit goals, D3 accounts for the luxuries and challenges of the datacenter environment. For instance, an important luxury is the fact that the datacenter is a homogenous environment owned by a single entity. Consequently, incremental deployment, backwards compatibility, and being friendly to legacy protocols are nongoals. The key insight guiding D3 design is the following: given a ows size and deadline, one can determine the rate needed to satisfy the ow deadline. Endhosts can thus ask the network for the required rate. There already exist protocols for explicit rate control wherein routers assign sending rates to endhosts [6,10]. With D3 , we extend these schemes to assign ows with rates based on their deadlines, instead of the fair share.

0). The router rst uses the packet header information to perform bookkeeping. This includes the ow returning its current allocation (line 3). Lines 7-13 implement the rate allocation scheme (Section 4.2) where the router calculates at+1 , the rate to be allocated to the ow. The router adds this to the packet header and forwards the packet. Snippet 1 Rate request processing at interval t Packet contains: Desired rate rt+1 , and past information rt and at. Link capacity is C. Router calculates: Rate allocated to ow (at+1 ). 1: //For new ows only 2: if (new ow ag set) N = N + 1 3: A = Aat //Return current allocation 4: D = Drt + rt+1 //Update demand counter //Calculate left capacity 5: lef t capacity = C A //Calculate fair share 6: f s = (C D)/N
7: if lef t capacity > rt+1 then 8: //Enough capacity to satisfy request 9: at+1 = rt+1 +f s 10: else 11: //Not enough capacity to satisfy request 12: at+1 = lef t capacity 13: end if
//Flows get at least base rate
14: at+1 = max(at+1 , base rate)
//Update allocation counter

15: A = A+ at+1

Of particular interest is the scenario where the router does not have enough capacity to satisfy a rate request (lines 11-12). This can occur in a couple of scenarios. First, the cumulative rate required by existing deadline ows, represented by the demand counter, may exceed the router capacity. In this case, the router simply satises as many requests as possible in the order of their arrival. In the second scenario, the demand does not exceed the capacity but fair share allocations to existing ows imply that when the rate request arrives, there is not enough spare capacity. However, the increased demand causes the fair share assigned to the subsequent rate requests to be reduced (line 6). Consequently, when the deadline ow in question requests for a rate in the next interval, the router should be able to satisfy the request.
4.4 Good utilization and low queuing
The rate given by a router to a ow is based on the assumption that the ow is bottlenecked at the router. In a multihop network, this may not be true. To account for bottlenecks that occur earlier along the path, 6
a router ensures that its allocation is never more than that of the previous router. This information is available in the rate allocation vector being carried in the packet header. However, the ow may still be bottlenecked downstream from the router and may not be able to utilize its allocation. Further, the veracity of the allocation counter maintained by a router depends on endhosts returning their allocations. When a ow ends, the nal rate request packet carrying the FIN ag returns the ows allocated rate. While endhosts are trusted to follow the protocol properly, failures and bugs do happen. This will cause the router to over-estimate the allocated rate, and, as a result, penalize the performance of active ows. The aforementioned problems impact router utilization. On the other hand, a burst of new ows can cause the router to temporarily allocate more bandwidth than its capacity, which results in queuing. To account for all these cases, we borrow from [6,10] and periodically adjust the router capacity based on observed utilization and queuing as follows: C(t + 1) = C(t) + (C(t) u(t) q ) ( ) T T

where, C is router capacity at time t, T is the update interval, u is bytes sent over the past interval, q is instantaneous queue size and , are chosen for stability and performance.4 Consequently, when there is under-utilization (u/T < C), the router compensates by allocating more total capacity in the next interval, while when there is queuing (q(t) > 0), the allocations reduce. Apart from addressing the downstream bottleneck problem, this ensures that the counters maintained by routers are soft state and divergence from reality does not impact correctness. The failure of endhosts and routers may cause ows to not return their allocation. However, the resulting drop in utilization drives up the capacity and hence, the allocation counters do not have to be consistent with reality. The router, during periods of low load, resets its counters to return to a consistent state. Even in an extreme worst case scenario, where bugs at endhosts lead to incorrect rate requests, this will only cause a degradation in the performance of the application in question, but will have no further eects in the operation of the router or the network.
in the datacenter. With a typical RTT of 300s, a new ow sending even just one 1500-byte packet per RTT equates to a send rate of 40Mbps! Most TCP implementations shipping today start with a send window of two packets and hence, a mere 12-13 new ows can cause queuing and packet loss on a 1Gbps link. This has been observed in past work [7,8] and is also present our experiments. With D3 , a new ow starts with a rate request packet with the SYN ag set. Such a request causes N to be incremented and reduces the fair share for ows. However, pre-existing ows may already have been allocated a larger fair share rate (N was less when the earlier rate requests were processed). Hence, allocating each new ow with its proper fair share can cause the router to allocate an aggregate rate larger than its capacity, especially when a burst of new ows arrives suddenly. We rely on D3 s ability to pause ows by assigning them the base rate to alleviate bursts. When a new ow starts, the fair share assigned to its rate request is set to base rate. For non-deadline ows, this eectively asks them to pause for a RTT and not send any data packets. The sender however does send a packet with only the rate request (i.e., a header-only packet) in the next RTT and the router assigns it with a fair share as normal. This implies that a new non-deadline ow does not make progress for an extra RTT at startup. However, such ows are typically long. Further, RTTs are minimal, and this approach trades-o a minor overhead in bandwidth and latency (one RTT 300s) for a lot of burst tolerance. Our evaluation shows that this vastly improves D3 s ability to cope with ow bursts over the state of the art. Additionally, this does not impact deadline ows much because the router still tries to honor their desired rate.

IMPLEMENTATION

4.5 Burst tolerance
Bursts of ows are common in datacenters. Such bursts are particularly challenging because of tiny RTTs
4 The , values are chosen according to the discussion in [10], where the stability of this controller was shown, and are set to 0.1 and 1 in our implementation. The update interval T should be larger than the datacenter propagation RTT; in our implementation it is set to 800s.
We have created an endhost-based stack and a proofof-concept router that support the D3 protocol. This paper focuses on the congestion control aspects of D3 but our implementation provides a complete transport protocol that provides reliable, in-order delivery of packets. As with TCP, reliability is achieved through sequence numbers for data packets, acknowledgements from receivers, timer-based retransmissions and ow control. On endhosts, D3 is exposed to applications through an extended Sockets-like API. The extensions allow applications to specify the ow length and deadline when a socket is created. The core logic for D3 , including the rate control scheme, runs in user space. We have a kernel driver that is bound to the Ethernet interface exposed by the NIC driver. The kernel driver eciently marshals packets between NIC and the user-level stack. The router is implemented on a server-grade PC and 7
Bit Offset Current fields Previous fields Feedback fields

Desired rate

24 Allocation Vector
Sender (1) SYN/RRQ (t) SYN/ACK/RRQ (t) (3) ACK/RRQ/data (t+1) data (t+1)

Receiver (2)

. Allocation Vector

Scale factor

Previous Desired rate Previous Allocation Vector
. Previous Allocation Vector

ACK/RRQ (t+1)

Feedback Allocation Vector

FIN/RRQ/data (t+n) (6)

Figure 6: Congestion header for rate request and feedback packets.

FIN/RRQ (t+n)

implements a shared buer, store and forward architecture. To be consistent with shallow buers in todays datacenters, the router has a buer of 128KB per NIC. The router uses the same kernel driver as the endhosts, except the driver is bound to multiple Ethernet interfaces. All incoming packets pass to a user space process, which processes and forwards them on the appropriate interface. The design of the kernel driver and user-space application support zero-copy of packets. Router overhead. To keep per-packet overhead low in the router, we use integer arithmetic for all rate calculations. Although each packet traverses the user-kernel space boundary, we are able to sustain four links at full duplex line rate. Specically, the average packet processing time was less than 1s (0.208s), and was indistinguishable from normal packet forwarding in user-space. Thus, D3 imposes minimal overhead on the forwarding path and the performance of our prototype leads us to believe that it is feasible to implement D3 in a commodity router. Packet header. The D3 request and rate feedback packet header is shown in Figure 6. The congestion header includes the desired rate rt+1 , an index into the allocation vector and the current allocation vector ([at+1 ]). The header also includes the allocations for the previous RTT so that the routers can update their relevant counters - the desired rate rt and the vector of rates allocated by the routers ([at ]). Finally, the header carries rate feedback to the destination - a vector of rates allocated by the routers for reverse trac from the destination to the source. All rates are in Bytes/s and hence, can be encoded in one byte; 1Gbps equates to a value of 125. The scale factor byte can be used to scale this and would allow encoding of much higher rates. The allocation vectors are 6 bytes long, allowing for a maximum network diameter of 6 routers (or switches). We note that current datacenters have three-tiered, tree-like topologies [20] with a maximum diameter of 5 switches. The allocation vectors could be made variable length elds to achieve extensibility. Overall, our current implementation im8

as non-deadline ows. We term this RCPdc since it is eectively RCP optimized for the datacenter.5 With RCPdc , the fair share is explicitly communicated to the hosts (i.e., no probe-based exploration is required) and it has been shown to be optimal in terms minimizing ow completion times [10]. Hence, it represents the limit for any fair share protocol. We were unable to obtain the set of Windows kernel patches for implementing DCTCP; yet, as explained above, our RCPdc implementation represents an upper limit for DCTCPs performance. We further contrast D3 against deadlinebased priority queuing of TCP ows. Priority queuing was implemented by replacing the Netgear switch with a CISCO router that oers port-based priority capabilities. Flows with short deadlines are mapped to high priority ports. Our evaluation covers the following scenarios: Flow burst microbenchmarks. This scenario reects the case where a number of workers start ows at the same time towards the same destination. It provides a lower-bound on the expected performance gain as all ows compete at the same bottleneck at the same time. Our results show that D3 can support almost twice the number of workers, without compromising deadlines, compared to RCPdc , TCP and TCP with priority queuing (henceforth referred to as TCPpr ). Benchmark trac. This scenario represents typical datacenter trac patterns (e.g., ow arrivals, ow sizes) and is indicative of the expected D3 performance with current datacenter settings. The evaluation highlights that D3 oers an order of magnitude improvement over out of the box TCP and a factor of two improvement over an optimized TCP version and RCPdc. Flow quenching. We evaluate the value of terminating useless ows that do not contribute to application throughput. Our results show that ow quenching ensures the D3 performance degrades gracefully under extreme load.

6. EVALUATION

We deployed D3 across a small testbed structured like the multi-tier tree topologies used in todays datacenters. The testbed (Figure 8) includes twelve endhosts arranged across four racks. Each rack has a top-ofrack (ToR) switch, and the ToR switches are connected through a root switch. All endhosts and switches are Dell Precision T3500 servers with a quad core Intel Xeon 2.27GHz processor, 4GB RAM and 1 Gbps interfaces, running Windows Server 2008 R2. The root switch is further connected to two other servers that are used as trac generators to model trac from other parts of the datacenter. For two endhosts in the same rack communicating through the ToR switch, the propagation RTT, measured when there is no queuing, is roughly 500s. The endhosts are also connected to a dedicated 48-port 1Gbps NetGear GS748Tv3 switch (not shown in the gure). We use this for TCP experiments. Our evaluation has two primary goals: (i). To determine the value of using ow deadline information to apportion network bandwidth. (ii). To evaluate the performance of D3 just as congestion control protocol, without deadline information. This includes its queuing and utilization behavior, and performance in a multibottleneck, multi-hop setting.

Flow burst microbenchmarks
6.1 Exploiting deadlines through D3
To evaluate the benet of exploiting deadline information, we compare D3 against TCP. However, TCP is well known to not be amenable to datacenter trac patterns. To capture the true value of deadline awareness, we also operate D3 in fair share mode only, i.e., without any deadline information and all ows treated 9
In this scenario, a host in each rack serves as an aggregator (see Figure 3) while other hosts in the rack represent workers responding to an aggregator query. This is a common application scenario for online services. All workers respond at the same time and all response ows are bottlenecked at the link from the racks ToR switch to the aggregator. Since there are only three hosts in
While the core functionality of RCPdc mimics RCP, we have introduced several optimizations to exploit the trusted nature of datacenters. The most important of these include: exact estimates of the number of ows at the router (RCP uses algorithms to approximate this), the introduction of the base rate, the pause for one RTT to alleviate bursts, etc.

Number of Senders

App Throughput (%)

60 D3 RCPdc TCPpr TCP 40

Moderate Deadlines

(a) Tight deadlines

D3 RCPdc TCPpr TCP 35 40
Figure 9: Number of concurrent senders that can be supported while ensuring more than 99% application throughput. each rack, we use multiple workers per host. We also repeated the experiment on a restructured testbed with more hosts per rack and the results remained qualitatively the same. The response ow lengths are uniformly distributed across [2KB, 50KB] and the ow deadlines are distributed exponentially around 20ms (tight), 30ms (moderate) and 40ms (lax). As described earlier, the primary performance metric is application throughput, that is, the number of ows nishing before their deadline. This metric was intentionally chosen as datacenter operators are primarily interested in the operating regime where the network can satisfy almost all ow deadlines. Hence, we vary the number of workers sending ows and across 200 runs of this experiment, determine the maximum number of concurrent senders a given congestion control scheme supports while ensuring at least 99% application throughput. This is shown in Figure 9. As compared to RCPdc , D3 can support almost twice as many concurrent senders while satisfying ow deadlines (3-4 times as compared to TCP and TCPpr ). This is because D3 uses ow deadline information to guide bandwidth apportioning. This is further pronounced for relaxed deadlines where D3 has more exibility and hence, the increase in the number of senders supported compared to tight deadlines is greater than for other approaches. Note that since congestion occurs at the aggregators downlink, richer datacenter topologies like VL2 [20] or FatTree [21] cannot solve the problem. For completeness, Figure 10 shows the application throughput as we vary the number of concurrent senders to 40. While the performance of these protocols beyond the regime of high application throughput may not be of primary operator importance, the gure does help us understand application throughput trends under severe (unplanned) load spikes. Apart from being able to support more senders while ensuring no deadlines are missed, when the number of senders does become too high for D3 to satisfy all deadlines, it still improves application throughput by roughly 20% over TCP, and 10

App Throughput (%) D3 RCPdc TCPpr TCP 35 40

(c) Lax deadlines

Figure 10: Application throughput for varying concurrent senders and tight, moderate and lax deadlines (the Y-axis starts at 60%). 10% or more, over RCPdc and TCPpr. Hence, even if we relax our operating-regime metric to be less demanding, for example, to 95% of application throughput, D3 can support 36 senders with moderate deadlines compared to the 22, 10 and 8 senders of RCPdc , TCPpr and TCP respectively. The gure also illustrates that RCPdc outperforms TCP at high loads. This is because probe-based protocols like TCP attempt to discover their fair share by causing queues to build up and, eventually, losses, ergo increasing latency for a number of ows. Instead, RCPdc avoids queueing by equally dividing the available capacity. Figure 11 highlights this point by displaying the scatter plots of ow completion times versus ow deadlines for TCP, RCPdc and D3 for one of the experiments (moderate deadlines, 14-30 senders). For TCP, it is evident that packet losses result in TCP timeouts and very high completion times for some ows. Since the hardware switch has a buer of 100KB, a mere eight simultaneous senders with a send-window of eight full sized packets can lead to a loss; this is why TCP performance starts degrading around eight senders in Figure 10.6
6 The mean ow size in our experiments is 26KB. Hence, a majority of ows will require more than 3 RTTs to complete,

D0 Number of Senders

Deadline (ms)
Loss followed by TCP timeout 10 100
Flows still miss deadlines 100
Flow completion time (ms)

App. Throughput (%)

Figure 11: Scatter plot of ow completion times vs. deadlines for TCP (left), RCPdc (middle), and D3 (right). Points above the diagonal reect ows that have met their deadlines. Fair share protocols, like RCPdc and DCTCP, address precisely this issue by ensuring no losses and short queues in the network; yet, as they are unaware of deadlines, a large number of ows still miss their associated deadlines (see middle Figure 11). For these particular experiments, RCPdc misses 8.1% of ow deadlines as compared to 1% for D3. Further, looking at ows that do miss deadlines, RCPdc causes their completion time to exceed their deadline by 75% at the 95th percentile (30% on average). With D3 , completion times are extended by 27.7% at the 95th percentile (9% on average). This implies that even if deadlines were soft, fair share protocols still suer, as a non-negligible fraction of ows exceed their deadline signicantly. This is not acceptable for datacenters where good performance has to be ensured for a very high percentile of ows. For TCPpr , we use two-level priorities. Flows with short deadlines are mapped to the higher priority class. Suchdeadline awarenessimproves its performance over TCP. However, it remains hampered by TCPs congestion control algorithm and suers from losses. Increasing the priority classes to four (maximum supported by the switch) does not improve performance signicantly. Simply using priorities with RCPdc will not help either. This is because deadlines can vary a lot and require a high number of priority classes while todays switches only support O(10) classes. Further, as discussed in Section 3, switch prioritization is packet-based while deadlines are associated with ows. Finally, bursty datacenter trac implies that instead of statically mapping ows to priority classes, what is needed is to dynamically prioritize ows based on deadlines and network conditions. D3 tackles precisely all these issues! With background ows. We repeat the above ow burst experiment by adding long, background ows. Such ows are used to update the workers with the latest data, and typically dont have deadlines associated

at which point the TCP send window will exceed 8 packets.
Figure 12: Number of concurrent senders supported while ensuring more than 99% application throughput in the presence of long background ows.
100 DRCPdc TCPpr 70 TCP 5 90
Figure 13: Application throughput with varying number of senders, 2.9KB response ows and one long background ow.
with them. For each run, we start a long ow from a trac generator to the receiver. This ow is bottlenecked at the link between the receiver and its ToR switch. After ve seconds, all senders send their responses ([2KB, 50KB]). Figure 12 shows the high application throughput operating regime for the three protocols. When compared to the response-ows-only scenario, the relative drop in the maximum number of senders supported is less for D3 than for TCP, TCPpr and RCPdc. Performance signicantly degrades for TCP and TCPpr in the presence of queuing due to background ows, which has been well documented in the past [5]. It is noteworthy that even with only two senders, TCP cannot achieve 99% of application throughput. TCPpr implements a 3-level priority in this scenario, where background ows are assigned to the lowest priority class, and deadline ows are assigned according to their deadline value to the other two priority classes. However, the background ows still consume buer space and hurt higher priority response ows. Hence, D3 is even better at satisfying ow deadlines in the presence of background trac. With tiny response ows. TCPs travails with ows bursts seen in datacenters have forced application designers to use various hacks as workarounds. This includes restricting the response ow size [5]. Here, we re11

Flows/s

TCP (reduced RTO) 100 (1100)

RCPdc 1300

D3 2000
Table 1: Flow arrival rate supported while maintaining >99% application throughput. peat the ow burst experiment with a uniform response size of 2.9KB such that response ows are 2 packets long. Further, there exists one long background ow. Figure 13 shows the application throughput with moderate deadlines (mean=30ms). With TCP, the long ow lls up the queue, causing some of the response ows to suer losses. Consequently, application throughput starts dropping at only 12 senders. As a contrast, the other three approaches satisfy all deadlines untill 40 senders. Since response ows are tiny, there is no room for D3 to improve upon RCPdc and TCPpr. However, our earlier results show that if designers were to be freed of the constraints imposed by TCP ineciencies, D3 can provide signicant gains over fair sharing approaches and priorities. Since RCPdc presents an upper limit for fair share protocols and further performs better than priorities, in the following sections, we will focus on comparing D3 against RCPdc only; TCP will also be presented as a reference for todays status quo.

6.2 D3 as a congestion control protocol
We also evaluated the performance of D3 as a congestion control protocol in its own right, operating without any deadline information. Results in earlier sections already show that D3 (and RCPdc ) outperforms TCP in terms of short ow latency and tolerance of ow bursts. Here we look at other aspects of D3 performance. Throughput and Queuing. We rst evaluate the behavior of D3 with long ows to determine the network throughput achieved. To this eect, we start a varying number of ows (2-20) to an endhost and record the ow throughput at 1ms intervals. The CDF for the aggregate throughput achieved by the ows is shown in Figure 16(a). The gure shows that the aggregate throughput remains the same as we increase the number of long ows with a median and average network throughput of 0.95Gbps (95% of capacity). Overall, D3 matches the performance oered by TCP for long ows. We also measured the instantaneous queue length on the bottleneck link to determine D3 s queuing behavior. 13

DISCUSSION

While we have evaluated many aspects of D3 design and performance, a number of issues were not discussed in detail due to space limitations. We briey comment on the most important of these here. Deployability. D3 takes a radical tact to align the datacenter network to application requirements. It mandates changes to almost every participant: applications, endhosts, and network elements. While in all cases these changes might be considered easy to implement, the choice of modifying the main components is quite intentional. Discussions with both datacenter operators and application designers have revealed that they are quite willing to adopt new designs that would free them from the articial restrictions posed by existing retrotted protocols. This is especially true if the added
benet is signicant as in the case of D3. The use of a UDP-based transport protocol by Facebook is a good example of the extra miles designers are willing to go to overcome current limitations [15]. The biggest hurdle to D3 deployment may be the changes necessitated to network elements. From a technical perspective, we have strived to restrict the state maintained and the processing done by a D3 router to a bare minimum. This allowed our user-space software implementation to easily achieve line-rates and bodes well for a hardware implementation. For instance, like RCP [10], D3 can be implemented on NetFPGA boards. Soft vs. hard deadlines. Throughout the paper, D3 operates under the assumption that deadlines are hard, and once missed, ows are useless. This decision was intentional to stress a, perhaps, extreme design point: Being deadline aware provides signicant value to the network. On the other hand, one can imagine applications and ows that operate on soft deadlines. Such ows may, for example, gracefully degrade their performance once the deadline is missed without needing to be quenched. The D3 model can accommodate soft deadlines in a number of ways. For example, since the host controls the requested rate, ows with soft requirements could extend their deadlines if these are missed, or fall back to fair share mode; alternatively, a two-stage allocation process in the router could be implemented, where demands are met in the order of importance depending on the network congestion. Yet, even with the presence of soft deadlines, the evaluation in Section 6 stresses the benets of deadline-aware protocols over fair share and priorities. Speculative ow quenching. Beyond the mechanisms described in Section 6.1.3, we are currently experimenting with more aggressive ow quenching heuristics, where hosts attempt to predict ows that will eventually miss their deadline. Such heuristics include, for example, successive increases of the desired rate over multiple RTTs. However, we believe that the aggressiveness of speculative ow quenching depends on the applications under consideration and we defer its discussion to future work.

[16] M. Welsh, D. Culler, and E. Brewer, Seda: an architecture for well-conditioned, scalable internet services, in Proc. of ACM SOSP, 2001. [17] D. Ferrari, A. Banerjea, and H. Zhang, Network support for multimedia: A discussion of the tenet approach, in Proc. of Computer Networks and ISDN Systems, 1994. [18] C. Aras, J. Kurose, D. Reeves, and H. Schulzrinne, Real-time communication in packet-switched networks, Proc.of the IEEE, vol. 82, no. 1, 1994. [19] B. B. Chen and P.-B. Primet, Scheduling deadline-constrained bulk data transfers to minimize network congestion, in CCGRID, May 2007. [20] A. Greenberg, J. R. Hamilton, N. Jain, S. Kandula, C. Kim, P. Lahiri, D. A. Maltz, P. Patel, and S. Sengupta, VL2: a scalable and exible data center network, in Proc. of ACM SIGCOMM, 2009. [21] M. Al-Fares, A. Loukissas, and A. Vahdat, A Scalable, Commodity Data Center Network Architecture, in Proc. of ACM SIGCOMM, 2008. [22] C. Guo, H. Wu, K. Tan, L. Shi, Y. Zhang, and S. Lu, Dcell: a scalable and fault-tolerant network structure for data centers, in Proc. of ACM SIGCOMM, 2008. [23] H. Abu-Libdeh, P. Costa, A. Rowstron, G. OShea, and A. Donnelly, Symbiotic routing in future data centers, in ACM SIGCOMM, 2010. [24] M. Alizadeh, B. Atikoglu, A. Kabbani, A. Laksmikantha, R. Pan, B. Prabhakar, and M. Seaman, Data center transport mechanisms: congestion control theory and IEEE standardization, in Proc. of Allerton Conference on Communications, Control and Computing, Sep. 2008. [25] Y. Gu, C. V. Hollot, and H. Zhang, Congestion Control for Small Buer High Speed Networks, in Proc. of IEEE INFOCOM, 2007.