Flawed Routers Flood University of Wisconsin Internet Time Server (2003)

Table of Contents

Figure 1 is simply a chart of inbound postulation to our field complete a 48 hr period, tuesday done thursday, May 13-15, 2003.

The first half of the chart shows emblematic postulation levels for our campus, with highest inbound packet rates of astir 40,000 packets-per-second. However, arsenic you tin see, our inbound packet-per-second complaint accrued dramatically starting May 14 astatine astir 8AM localtime, chiefly from our commodity Internet Service Provider, WiscNet. At astir 9:40AM this further postulation began to origin problems pinch our measurement infrastructure and immoderate of our bequest intra-campus routers. By 11AM we had identified the inbound flood postulation by protocol and port numbers. It was destined for our nationalist clip server and we blocked the incoming postulation upstream, astatine WiscNet's separator routers, which alleviated the problem for the clip being. This is simply a emblematic action for network operators to return successful guidance to malicious Denial-of-Service flood attacks, of which we assumed this was one.

The postulation successful mobility appeared to beryllium Network Time Protocol (NTP) queries in that they consisted of 76-byte IP packets destined for UDP larboard number 123 (NTP). However, these packets had an different characteristic: though they appeared to travel from galore sources, they each had the aforesaid root larboard number: 23457. Therefore, it was imaginable to configure our routers to artifact conscionable a subset of inbound queries to our NTP server, and proceed to work the other legitimate requests normally. We conscionable blocked each UDP postulation sourced from larboard 23457 and destined for larboard 123 (NTP) of the NTP server in question. (Note that the number 23457 seems hand-picked, arsenic the number subsequent to 23456.) At this constituent we simply chalked it up to naivete connected the portion of the "attacker", which we presumed was forging many random root addresses, and near it astatine that, presuming that the flood would subside wrong hours arsenic "script kiddie"-launched flood attacks often do.

Paraphrased from RFC2030 by Dave Mills:

Note that SNTP uses the aforesaid packet format arsenic NTP. In this way, SNTP clients tin utilize NTP servers, moreover though they do not instrumentality the complexities of the afloat peer-to-peer NTP protocol.

SNTP conversations typically travel these steps:

1. A customer that would for illustration to cognize the clip sends a UDP packet containing the SNTP petition to the well-known NTP larboard number 123 of an NTP server, and awaits a reply.

   Figure 2. A SNTP Request Packet

   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |LI | VN |Mode | Stratum | Poll | Precision | | =0|= 1-4|= 3 | = 0 | = 0 | = 0 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Root Delay | | = 0 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Root Dispersion | | = 0 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Reference Identifier | | = 0 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | | Reference Timestamp (64 bits) | | = 0 | | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | | Originate Timestamp (64 bits) | | = 0 | | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | | Receive Timestamp (64 bits) | | = 0 | | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | | Transmit Timestamp (64 bits) | | = n | | (some number: zero, aliases the clip of petition sent by client) | | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

2. The server responds pinch a UDP packet containing the SNTP reply from the well-known NTP larboard number 123 to the SNTP client.

   Figure 3. A Unicast SNTP Reply Packet

   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |LI=| VN |Mode | Stratum | Poll | Precision | |0-2|=req.|= 4 | = 1 - 14 | (ignore) | (ignore) | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Root Delay | | (ignore) | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Root Dispersion | | (ignore) | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Reference Identifier | | (ignore) | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | | Reference Timestamp (64 bits) | | (ignore) | | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | | Originate Timestamp (64 bits) | | (copied from petition Transmit Timestamp) | | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | | Receive Timestamp (64 bits) | | (time petition was received by server) | | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | | Transmit Timestamp (64 bits) | | (time of reply sent by server) | | = n | | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

3. Upon receiving the response, the customer optionally uses the Originate Timestamp from the reply to validate the response, attempting to assure that it is so a consequence to this client's request. (If the reply were spoofed from different source, it would beryllium improbable to incorporate the correct worth arsenic the Originate Timestamp). Then it plucks the value from the "Transmit Timestamp", possibly modifying it somewhat to account for the estimated one-way end-to-end delay, and uses the result arsenic the existent clip to group its section clock.

Now, backmost to our story...

One period later, we discovered that the flood of inbound NTP traffic persisted astatine an moreover much incredibly precocious rate, arsenic evidenced by Figure 4 which land our router's discarded packet rates opening successful early June 2003, for the postulation successful question.

In Figure 4, statement that: (1) location are flimsy regular fluctuations successful rate (perhaps owed to diurnal personification behavior), (2) generally, the complaint stays at a worth complete 250,000 packets-per-second (and complete 150 megabits-per-second), and (3) that the postulation complaint increases throughout the clip shown. The crisp drops successful postulation complaint successful this figure are not owed to the flood subsiding but alternatively were owed to network attraction and a impermanent artifact upstream from the observation point.

Once we recovered that this flood was continuing and was still expanding in rate, we investigated further. By cautiously removing the artifact connected some ingress interfaces, we allowed a trickle of postulation done to the server and captured the packets including their payload. We learned that these packets appeared to beryllium legitimate, well-formed Simple Network Time Protocol (SNTP) type 1 queries, albeit astatine an inexplicably precocious complaint from each customer host. For instance, during one trace, galore clients produced astir 1 query per second. This would be highly different for a decently constructed SNTP client, since an application which uses SNTP is simply willing successful mounting its own clock comparatively accurately truthful that its big has immoderate reasonable notion of the existent time. One query per 2nd is ridiculous, and is far from champion believe for NTP customer behavior.

Also, we discovered that galore of the IP addresses could beryllium resolved to DNS names and moreover that the IP addresses each appeared to be valid sources for the fixed ingress interface from which we removed the block. This indicated that it was rather imaginable that the source addresses were not forged but alternatively were existent Internet hosts running immoderate very different SNTP client.

Alas, nary of the customer root hosts were wrong our section campus network. This meant we would request to enlistee the thief of unit at remote sites to assistance successful the investigation.

Of the apical talker root IP addresses from the aforementioned packet trace, I selected 2 customer hosts from different universities pinch talented network unit who would beryllium acquainted pinch responding to specified incidents.

The pursuing is an email I sent to the Incident Response Team astatine 1 of those institutions to which 1 of the customer big addresses belonged. (To support a modicum of anonymity, I person replaced the existent SNTP client's IP reside pinch 10.42.69.10 and person besides removed the email domain names.)

Network unit from 2 universities investigated 2 root hosts which I reported arsenic being sources of these anomalous SNTP queries. Both reported that a Netgear marque router was the root of the traffic. (Specifically, one was identified arsenic exemplary MR814.)

Now things started to make immoderate sense. Many root hosts each utilizing the same source larboard number could beryllium explained by an embedded SNTP customer successful which the programmer hard-coded the root larboard number (23457).

While searching the web for inheritance accusation connected Netgear products' acclaimed NTP support, I came crossed the pursuing quote (from ICSA Labs Firewall Lab Report connected the NETGEAR FR114P):

Conclusion

The Candidate Firewall Product met each the criteria elements successful the Baseline and Residential modules and truthful has attained ICSA Labs Firewall Certification.

In bid to verify our presumption that the root of the flood of SNTP queries are the Netgear Platinum family products and to properly characterize this problem to the vendor, the Netgear codification for a number of their products was downloaded and investigated.

Simply by utilizing the Unix "strings" command, I was capable to verify that indeed the Netgear codification seems to incorporate the magic number 23457 (as a larboard number):

Using a akin technique, I recovered the pursuing IP addresses embedded as ASCII strings successful RP614_4_12.bin:

Of 3 globally routable IP addresses therein, only 128.105.39.11 appears to beryllium utilized arsenic an NTP server. One of the others was an IP address antecedently utilized by the "dyndns.org" move DNS sanction service. Netgear has reported to america that the remaining embedded globally routable IP addresses are nary longer utilized and that they are portion of dead code near complete from debugging by 1 of the developers.

On June 16, 2003 I sent the pursuing email connection to Netgear support. Since this rumor is much important than the emblematic customer support inquiry, I besides sent it straight to immoderate Netgear employees (whose email addresses were culled from the web) asking them to communicate it to the due group successful engineering and/or to have someone interaction maine by telephone aliases email.

After receiving nary consequence for days, I called Netgear's headquarters, leaving messages pinch 2 executives explaining the seriousness of the situation. I besides emailed members of Netgear's executive squad by guessing their email addresses, based upon their email naming convention. I included a "Return-Receipt-To" header, and their Mail-eXchanger notified maine that each were delivered successfully. Here's a information of that message:

I perfectly request to perceive from responsible parties astatine NETGEAR immediately, if NETGEAR wishes to statesman a speech earlier this goes public. We're not expecting an contiguous solution; successful fact, I'm fairly certain location is nary complete solution without UW-Madison's involvement.

Netgear's support statement was wholly unresponsive. Curiously, I did yet person the email connection beneath from Netgear's email-based customer support system, immoderate 23 days aft I submitted the problem study connected June 16.

Shortly aft opening a speech pinch Netgear, I projected the formation of a reappraisal squad to talk imaginable solutions. Netgear agreed, and a reappraisal squad was formed pinch astir 15 members, a third from each of these areas:

• Netgear employees
• University employees
• Independent experts from their respective fields:
• Regional Internet Registries
• Internet Measurement Research
• Network Time Protocol

A number of action items and directions were developed during the review process. These included:

• Fix the SNTP Client
• Propose the Network Operational Options
• Inform the Internet Community
• Clarify Internet Best Current Practice and Protocol Standards

The Flawed Netgear SNTP Client implementation successful the products affecting UW-Madison has the pursuing characteristics:

• Uses a hard-coded IP reside for the NTP server 128.105.39.11, that of ntp1.cs.wisc.edu.

• Uses a fixed UDP root larboard number 23457.
  This was incredibly advantageous arsenic it allowed UW-Madison to identify and count the Netgear clients. However, owed to the wide usage of Network Address Port Translation (NAPT, aliases NAT/PAT) upstream from immoderate Netgear products, the SNTP petition root port number is sometimes rewritten earlier the petition packet reaches its destination.

  Note to web operators: Please do not artifact UDP traffic involving larboard 23457 nor postulation involving our NTP server's IP address of 128.105.39.11. While we admit attempts to help, it may interfere pinch the champion imaginable solution to this problem.

• Polls astatine one second intervals until it receives a response from the NTP server, aft which it uses a longer canvass interval specified as one minute, 10 minutes, 2 hours, aliases 24 hours, depending upon merchandise exemplary and firmware version.

As of this penning (August 2003) the University is making its best effort to work the Netgear clip requests. As such, users of the affected products should not usually announcement immoderate problems owed to this flaw. Furthermore, based connected acquisition truthful far, it seems that only a mini subset of the customers are moreover alert of the time-related features of these products (which see logging, argumentation scheduling, and email notifications).

In parallel, Netgear has produced and continues to activity connected firmware that does not grounds the aforementioned problems. Customers can upgrade to newer firmware versions, which are disposable for download from Netgear's support site. At the clip of this penning (August 2003), the astir existent type of firmware disposable for the RP614v2, RP614, DG814, and MR814 models does not utilize UW-Madison's time service nor does it canvass excessively frequently.

I person counted much than 500,000 unsocial Netgear sources that queried our time server successful 1 day. This measurement apt underestimates the actual count because of Network Address Port Translation, which modifies the source IP reside and larboard number, and because immoderate broadband residential services driblet the customer's nexus erstwhile the work is not successful use.

As of June 30, 2003, Netgear reported a full of 707,147 affected products manufactured. Some elemental math: If location are 700,000 errant SNTP clients each of which tin make 1 SNTP petition per 2nd to our clip server, past the worst-case aggregate complaint will beryllium about 700,000 packets per second. Since each SNTP packet is 76 bytes in size, that is besides 426 megabits per 2nd of traffic.

Figure 8a shows the existent number of unsocial NTP Netgear customer IP addresses observed per time by a router connected UW-Madison's network. Theoretically, counting the clients successful this measurement could overestimate the count if the clients' DHCP servers changes the customer IP address frequently. However, based connected the number of products reported arsenic having been manufactured, it seems reasonably accurate.

• SHOULD usage a canvass interval wrong the scope from 64 to 1024 seconds aliases longer
• SHOULD usage section NTP server(s) aliases multicast erstwhile available, as configured by the usability aliases wished by a find mechanism specified arsenic via the DHCP "Network Time Protocol Servers Option", which is defined successful conception 8.3 of RFC 2132.
• MAY capacity exponential backoff of canvass interval (within the aforementioned range) upon nonaccomplishment to person a consequence from the NTP server(s)
• MUST NOT usage a shorter canvass interval upon nonaccomplishment to person a consequence from the NTP server(s)
• MUST let the usability to configure the query behaviour pinch respect to whether aliases not it is enabled aliases abnormal and pinch respect to which campaigner clip servers tin beryllium queried.
• SHOULD usage the Domain Name System to find campaigner server(s) IP address(es), truthful that the NTP server's area administrator can power the customer behavior.
• SHOULD resoluteness the server IP reside via DNS earlier each poll/query, truthful that the pertinent DNS entries' Time-To-Live values are respected.
• SHOULD support the existing NTP access-control system by, upon receiving a valid `kiss-of-death' packet, reporting the condition and discontinuing queries to the server successful mobility until reinitialization.
• MAY usage an implementation-defined fixed root larboard number

During the reappraisal process, we learned that Netgear already was having SNTP-related codification changes developed for the RP614v2 merchandise anterior to my initial notification of the problems the flaw was causing to the University.

Regarding Firmware v5.13 RC7 for the RP614v2, Netgear made this new code disposable to maine connected July 10. My testing recovered that the modified SNTP customer had these characteristics, overmuch arsenic they described:

• Now requires a DNS server to beryllium configured (or learned via DHCP) before generating immoderate SNTP queries.
• The codification performs DNS queries for "time-a.netgear.com" and "time-b.netgear.com" astatine 10 infinitesimal intervals until success, alternating names if nary consequence is received. I verified besides that it supported responses pinch CNAMEs aliases aggregate A records arsenic well.
• Following successful DNS resolution, it sends an NTP query to the resolved IP reside and waits for a reply. If nary reply comes successful 10 minutes, it again resolves the name, and requeries. It appears to springiness up aft five retries.
• Whenever immoderate configuration alteration is applied via the web interface, it causes the device's timepiece to beryllium zeroed, the NTP server to beryllium re-resolved, and subsequently queried.

• The SNTP customer successful this codification does not look to validate the NTP response packet. It will judge immoderate incoming packet to port 23457 arsenic a valid consequence moreover if the flags are group incorrect (for instance, indicating that it is different customer query alternatively than a server response).
• While the SNTP customer is awaiting a consequence (after querying either time-a aliases time-b) it seems to judge immoderate UDP response packet, moreover if the root IP reside of that UDP packet is not that of the clip server that it queried.

Netgear continues to create improvements to their SNTP customer and has vetted the creation pinch the reappraisal team.

These flawed devices are not easy reconfigurable. Representatives from some Netgear and UW-Madison judge that it is not a viable option to trust connected Netgear's customers to upgrade to the newer firmware (the first of which was released successful July) to correct the errant behavior.

Our reappraisal squad has considered a number of imaginable options astir how to woody pinch the errant Netgear clip requests. While I won't discuss all the specifications here, the 2 chiefly endgames connected which we've focused are outlined below.

In this action we would deploy highly-reliable, redundant NTP servers at WiscNet's borders and way the inbound requests destined to 128.104.39.11 to them utilizing BGP anycast. (Anycast is a technique that tin often beryllium employed to way postulation for some stateless RPC services, specified arsenic DNS aliases NTP, which are based upon UDP.) Implementing this action would apt see placing a brace of rack-mount NTP servers astatine each of 3 locations wrong WiscNet: UW-Madison, UW-Milwaukee, UW-Eau Claire. These are nearest the three current separator Internet speech points and truthful supply the most diverse paths for reliability of connectivity to the world Internet.

One chopped advantage of this configuration is that UW-Madison retains arsenic overmuch power arsenic imaginable complete its precious IPv4 address allocations. Because this BGP anycast deployment resides solely within WiscNet (which will grant a azygous /32 host-address route), this option consumes arsenic small of UW-Madison's IP reside abstraction arsenic imaginable - just the reside to which Netgear clip requests were directed.

Endgame A has immoderate risk. Whether aliases not the servers' responses reach the requesting customer big is not wholly wrong the University's control, result immoderate magnitude of flooding will apt continue. There are galore reasons different than server nonaccomplishment for disruptions successful the end-to-end way betwixt the SNTP clients and servers that could cause the clients not to person the responses and to flood requests toward our servers anyway. These see asymmetric routing problems, firewalling policies, and disasters affecting immoderate nexus betwixt the clients and servers. Indeed, moreover while our clip server is dutifully responding to each netgear SNTP requests, we still regularly observe that hundreds of them proceed to flood. Apparently these "zombies" never person our responses.

To limit the anticipation of the aggregate servers being simultaneously isolated from the Internet, 1 could see an moreover more geographically divers group of deployment locations, specified arsenic that done by the AS112 Project, which effectively mitigates the harm caused to the Internet's guidelines name servers by RFC1918-related queries.

Figure 10 is simply a sketch showing really this work would work. The Netgear SNTP requests heading toward UW-Madison are shown successful green. Note that aggregate NTP servers, each pinch the aforesaid IP address, are located successful aggregate locations. WiscNet's separator routers divert the inbound SNTP requests to the nearest server. The server responses are shown successful red. If immoderate of the servers fail, the postulation should route to 1 of the remaining NTP servers pinch the aforesaid address.

To forestall Netgear clip requests from being forwarded to our network would require UW-Madison to sacrifice a artifact of IP reside space within the people B web which includes the IP reside of ntp1.cs.wisc.edu.

Because of the measurement the Internet's backbone routing is operated, and to keep the number of routes manageable, web routes are sometimes not respected unless they are sufficiently large. In today's Internet, that intends a way mightiness not beryllium considered morganatic unless it represents 2,048 aliases 4,096 contiguous addresses. Respectively, network operators would telephone those size "/21" aliases "/20" (pronounced "slash twenty") blocks because they correspond networks having netmasks of 21 or 20 contiguous bits.

Figure 11 is simply a sketch showing this configuration. The BGP updates originating from UW-Madison's separator router are shown successful red. The Netgear SNTP Requests are shown successful green. The ICMP unreachable messages returned to the customer by BGP-aware separator routers passim the Internet are shown in blue. These pass the customer that the web successful which the NTP server would reside is unreachable.

This endgame that tries to suppress the forwarding of requests comes at a important costs to the University - we whitethorn person to sacrifice, likely for the life of the flawed products, arsenic galore arsenic 4,096 IP addresses. Figure 12 shows really our existing 128.105.0.0/16 network could beryllium divided, and the 1 portion "/20" artifact which would beryllium excluded from the Internet's world BGP routing table.

The risks of endgame B see the anticipation that immoderate portions of the Internet mightiness not beryllium capable to scope morganatic field IP addresses that lie adjacent the sacrificial, unadvertised block. Input from the backbone network operations organization and real-world acquisition must determine which solution champion serves the University and Internet organization arsenic a whole.

The nationalist merchandise of this archive is portion of an effort to pass the Internet organization of this flaw and the resulting floods, pinch the hope of minimizing the likelihood of specified a correction being repeated elsewhere. Note that Netgear was notified of our scheme to publicly disclose the specifications herein weeks successful beforehand of its release. Furthermore, earlier revisions of this archive were provided to them for review.

Because of the scope of the resulting problem, pinch hundreds of thousands of ill-behaved Internet hosts distributed world-wide, and because of the standard and unexpected quality of the flooding, with aggregate rates which could wholly capable immoderate web links, I felt that it was important to pass others and solicit proposal from experts.

Following this disclosure, it's my intent to find due venues to further coming and reappraisal the dangers and imaginable solutions to this and akin problems.

For instance, during the reappraisal process we learned that the Commonwealth Scientific & Industrial Research Organisation (CSIRO) successful Australia is having similar problem pinch astir 85,000 SMC marque routers that canvass the CSIRO time server doubly a infinitesimal erstwhile they don't person a response. A communicative astir that incident, "Rogue routers origin havoc for CSIRO", tin beryllium recovered here:

While the standard of the CSIRO problem is orders-of-magnitude less, with floods of possibly 2,800 packets per 2nd and 1.7 megabits per second, it is strikingly akin and possibly not arsenic apt to beryllium as responsibly addressed pinch the assistance of that manufacturer.

Also during the reappraisal process, immoderate members of the reappraisal squad began work connected Internet Drafts to amended archiving pertinent to this issue. There are astatine slightest 2 specified efforts presently successful their infancy:

1. I americium successful the process of preparing an Internet Draft, currently titled "Embedding Globally Routable Internet Addresses Considered Harmful", which denounces the believe of embedding unique, globally routable IP addresses successful Internet hosts, describes immoderate of the resulting problems, and considers selected alternatives.
2. Members of the NTP organization person revised and reviewed the existing Informational RFC2030 that describes SNTP. Through their efforts and perhaps those of different willing parties, it whitethorn beryllium imaginable to revisit NTP and SNTP arsenic a standards way protocol wrong the IETF.

I'm pleased to study that Netgear has cooperated pinch america connected the initial steps of this process and we are forging an statement that will enable us to instrumentality a suitable solution.

For the clip being, UW-Madison continues to work Netgear SNTP requests successful spite of receiving occasional large-scale floods of traffic from Netgear products. A caller incident is shown successful Figure 13. The shark-fin shaped anomaly connected the correct is simply a flood of inbound Netgear clip requests which grew to astir 100,000 packets per second before subsiding.

Both the magnitude and long of the Netgear-caused incidents continue to coming a superior operational problem for UW-Madison. While fundamentally progressive successful a crippled of state roulette astatine the moment, we are hoping to utilize the expertise of some UW-Madison and the Internet usability organization to creation and instrumentality a bully solution.

• What does this unintentional Denial-of-Service flood bespeak astir the viability of immoderate nationalist Internet services?
• Can the Internet routing infrastructure beryllium improved to alteration little disruptive solutions to specified problems?
• Are incidents specified arsenic this a apt side-effect of ubiquitous, low-cost, perhaps moreover disposable Internet hosts?
• Are the manufacturer, vendor, Internet operations, and personification communities willing and capable to cooperate to reside specified problems?

The pursuing provided assistance pinch the information gathering, analysis, and group networking:

• University of Wisconsin-Madison: Jeff Bartig, Jim Gast, Michael Hare, Adam Kunen, Dave Thompson
• University of Florida: Robert Bird, Greg Goddard
• Harvard University: Greg Mazzu
• k claffy, Nevil Brownlee, George Michaelson

I'd besides convey the members of the reappraisal team, including those remaining anonymous. I'm definite we'll travel to a amended solution because of their participation.

• MRTG
• RRDTOOL
• RRGrapher
• junipoll
• flow-tools
• Cflow
• FlowScan
• tcpdump
• ethereal (now wireshark)
• sntp.pl
• strings ;^)

1. RFC2030: Simple Network Time Protocol (SNTP) Version 4 for IPv4, IPv6 and OSI (status: Informational)
2. RFC1305: Network Time Protocol (Version 3) Specification, Implementation and Analysis (status: DRAFT STANDARD)
3. home of the Network Time Protocol (NTP) project
4. Public NTP Time Servers
5. Public NTP Secondary (stratum 2) Time Servers
6. RFC2132: DHCP Options and BOOTP Vendor Extensions
7. RFC1546: Host Anycasting Service
8. http://www.networksorcery.com/enp/protocol/sntp.htm
9. an article connected really to alteration Windows' clip server configuration utilizing regedit
10. Basic Operation of the Windows Time Service
11. http://www.microsoft.com/windows2000/docs/wintimeserv.doc
12. Flawed Routers Flood University of Wisconsin Internet Time Server (NANOG 29 presentation)
13. A Case Study successful Internet Pathology: Flawed Routers Flood University's Network (LISA '03 talk)

1. What is Netgear's liability for causing (however inadvertently) this denial of work for your network?

   My activity responsibilities are not ones that would make maine a subordinate successful such negotiations. However, arsenic I reported, an statement is being forged.

   Note to others: Please do not inquire maine for financial aliases ineligible details. This archive is simply a method summary of the business and is not the vehicle by which to present specified details.

   You whitethorn beryllium willing successful this news article: http://www.doit.wisc.edu/news/story.asp?filename=322

2. Have you considered putting up a server which sends backmost clone answers to netgear clients, to origin group to upgrade?

   Yes, but we didn't see it for long. Both Netgear and others on the reappraisal squad agreed that it would likely have very small effect, since only a mini subset of the customers seem to moreover beryllium alert of the NTP-related features of the affected products. Besides, the projected "dishonest" clip server (which would report the incorrect time) would person to beryllium operated astatine the IP reside that is currently that of our well-known reliable clip server. It would be quite rude for it to abruptly go unreliable since the full purpose of the nationalist clip work is to reply pinch the correct time. The University intends to supply the champion imaginable service, sloppy of how it's abused.

3. What is the expected life-time for these products?

   Personally, I deliberation its apt that galore of them will beryllium astir until five to 10 years from now. Its only a guess, but possibly the half-life is five years, aft which we'd expect to spot little than 350,000 of the affected products remaining successful use.

4. In fig 13, could that "shark fin" spike person thing to do pinch past week's powerfulness grid nonaccomplishment (Blackout 2003), and consequent "rolling" restoration?

   No, it did not coincide pinch the blackout that affected the eastbound coast in August 2003. However, I did look for grounds of the blackout successful the netgear SNTP traffic, and recovered only flimsy little during the outage. Relatively small of the Internet was really affected by the powerfulness outage. Early reports were that only a fewer 1000 BGP prefixes from only a mates 100 autonomous systems were offline. Those that were offline were unreachable successful both directions, truthful we didn't person requests from them until powerfulness was returned. The rolling restoration served to administer the load to our server as those netgear clients came online.

5. Are location different devices than those mentioned which besides suffer from the flaw which causes inadvertent flooding of your network?

   A number of Netgear users person reported to maine that Netgear exemplary RO318, firmware type V3.26, besides utilizes our clip server and besides logs ntp errors to its information log. I person not evaluated this product, truthful do not cognize what retry times it uses, but will study this to Netgear.

6. What was the effect of this article being slashdotted?

   While having a important effect connected the web server, wide it was an insignificant level of postulation for the campus. The server handled 30 requests a 2nd for a while.

   Here's a chart comparing the web server hits-per-second to the number of Netgear-sourced SNTP petition flows-per-second, erstwhile this study was slashdotted. (Note that astir Netgear clients were receiving replies from our server astatine this time, truthful they were not flooding requests.)

   

7. Why is simply a accepted shaper recall/defect solution not a possibility?

   Both Netgear and different members of the reappraisal squad felt that it was unlikely that each but a very mini subset of the owners would return the affected device since they look to beryllium moving fine. Also, very fewer customers have registered these products pinch the manufacturer, truthful it is impractical to interaction them.

8. I'm pinch [the IT press], do you person immoderate clip to speak pinch me?

   When this study was first presented, an astute assemblage member proposed that the IT property ostensibly plays an important domiciled in evaluating whether aliases not user products comply pinch Internet standards and champion existent practice.

   Given that, if you're pinch the IT property and your publication does Internet merchandise reviews aliases makes "Editor's Choice" awards, I am willing to correspond pinch you by email astir this story. Specifically, the Internet organization whitethorn use from our exploring the IT press' familiarity pinch Internet standards and champion practices and the press' appraisal of its expertise to measure products.

   By tying the reporting of this communicative successful pinch the merchandise information and recommendation usability of the IT press, I'm hopeful that the community could trim the likelihood of specified flaws causing specified problems. It's conscionable an idea, fto maine cognize what you think.

9. How has this communicative been covered successful the press?

   While surely not an exhaustive list, present is simply a sampling of coverage.
   (Please beryllium alert that immoderate of these articles are misleading aliases incorrect astir the details;
   Only a fewer of them contacted maine to cheque the facts.)
   • Slashdot: Netgear Routers DoS UWisc Time Server
   • The Inquirer: Wisconsin Uni page describes Netgear `flawed routers'
   • Wisconsin Technology Network: Flawed Routers Flood UW Server
   • ZDNet: NetGear flaw triggers accidental DoS attack
     
   • CNET News.com: NetGear flaw triggers accidental DoS attack
   • ZDNet Australia: UPDATE: Netgear routers onslaught university
   • Network World: Netgear router quirk perturbs college
     (also successful people Network World magazine)
   • TECHWORLD: NetGear sets disconnected denial of work attack
   • Network Computing: Industry Insights: Opportunity Knocks / Quality Counts
     
   • PC World: NetGear Routers Wage War connected University
     (also successful people successful the November 2003 rumor of PC World magazine)