Go to Making Light's front page.
Forward to next post: I love my country
Subscribe (via RSS) to this post's comment thread. (What does this mean? Here's a quick introduction.)
The purpose of this entry is its comment thread. Read on.
Our mail is down, and yours may be too.
Post messages here if you want to get in touch with us, or if your ISP is affected and you’re trying to get in touch with someone who might turn up here, or if you want to swap information about the DDOS attacks.
If things get too lively and numerous I may open further threads, but for now this one is it.
What’s happening:
As I said, our mail is down. Patrick’s posted about this briefly in Electrolite. Panix, our ISP, is getting hit with a massive DDOS (distributed denial of service) attack. Sayeth Panix:
Many Panix services were disabled on-and-off (mostly off) from about 12:30AM Saturday morning up until recently. This was the result of a massive DDOS (distributed denial-of-service) attack that apparently affected a number of ISPs. We don’t know a lot about this yet, though we’ve been working on it all night, since the volume was far more massive than any attack in our previous experience, and apparently triggered at least two separate bugs in Cisco’s IOS (memory leakage and HSRP failures).See the first comment in the thread for a further description of what’s going on.The attack is continuing, on and off. We’ve taken certain measures, which are partially effective, but we’re uncertain as to how they’ll stand up.
This is disturbing. Panix has previously gotten hit with some state-of-the-art DDOS attacks. This one must be huge.
So that’s where things stand. To repeat what I said earlier:
Post messages here if you want to get in touch with us, or if your ISP is affected and you’re trying to get in touch with someone who might turn up here, or if you want to swap information about the DDOS attacks.
If things get too lively and numerous I may open further threads, but for now this one is it. (The other threads in my weblog are still the other threads. Simple.)
If Making Light slows down, please just be patient. If it stops working entirely, you can assume the attacks have spread to encompass our DSL provider and/or Blogomania. I’m hopeful about our chances of staying up, but there is that possibility. Make talk while the sun shines.
Addendum: A lot of sites out there will explain what DDOS attacks are and how they work, but if you’re starting from scratch, you might find it helpful to read Steve Gibson’s The Strange Tale of the Denial of Service Attacks Against GRC.COM, which tells the story of a series of attacks launched against Gibson Research Corporation in January 2002. It’s lively reading, and a good way to get a sense of the issues.
Don't know about a DDOS. I *do* know that 1) There is a massive MS-SQL 2000 worm running rampant on the net. If you have a MSSQL2K box on the net that doesn't have ports 1434 and 1433 firewalled, and isn't running at least Service Pack 2 for MSSQL2K, you are already compromised. Congratulations.
I'm personally seeing 30-50 attempts a minute on my home network. Plus, there is a huge amount of traffic going to port 137 (the MS NetBios) port, which may be related, or may be something else.
H D Moore on Bugtraq has already run the thing down.
A worm which exploits a (new?) vulnerability in SQL Server is bringing the core routers to a grinding halt. The speed of the propagation can be attributed to the attack method and simplicity of the code. The worm sends a 376-byte UDP packet to port 1434 of each random target, each vulnerable system will immediately start propagating itself. Since UDP is connection-less, the worm is able to spread much more quickly than those using your standard TCP-based attack vectors (no connect timeouts).Some random screen shots, a copy of the worm as a perl script, and a disassembly (sorry, no comments) can be found online at:
http://www.digitaloffense.net/worms/mssql_udp_worm/
The effect on the internet backbone providers is dramatic -- UUnet's latency is huge, several people are reporting dropped packets in the 90%+ range.
My email seems to be slowing to a trickle, which implies mvp.net may be getting caught in the crossfire.
Oops. It's service pack 3, not 2. The fix is also available at
http://support.microsoft.com/default.aspx?scid=kb;en-us;323875
It's dated 18-DEC-2002. It is 25-JAN-2003. Why haven't you patched?
Thank you, Erik Olson, Making Light's favorite BOFH. Do you want me to just incorporate that correction and the addenda into the original message? Your summary is likely to get passed around, and it'll travel better as a single parcel.
Meanwhile: Good morning! Isn't this vexing? Have a cup of coffee.
Good morning, John. Briefly, how's your ISP? Any rough Internet weather out your way?
Very vexing. We hope it's resolved soon. I think I'll have another cup of coffee, and go play on Salon's Table Talk, which is up.
Can't hit my blog. Vexing indeed. It's like not being able to get into your own kitchen in the morning. Glad Making Light is still here. Sort of like wandering down the street and finding the neighborhood bakery still open.
Glad to see you, Greg. Your weblog is definitely inaccessible. Do you still have e-mail?
Hi, Teresa. Yep, still have e-mail.
Wow. This whole thing is really messing with my morning routine. Crummy vandals.
Sleepless in Eugene...
It's still going on. UUnet is essentially down. I can't reach Panix or CERT at all.
Best short coverage.
AP article. Good worldwide coverage, with obligatory incorrect remark from gummint representative.
Well, it's now 11:04 am and www.panix.com seems back and I was just able to check email and get onto the panix newsgroups. Though my email just dinged at me indicating it had lost the connection again. Some progress, anyway, I guess.
Michael! Heya.
Patrick had one letter trickle through, but the last mail in my inbox is still from around midnight.
Hey, Greg. Wondered what was up with your neck of the woods.
Heya T.!
Yeah, I imagine they have some catching up to do with mail, news, etc. I just sent a message to my panix account from a yahoo.com account and it came through in an impressively jaunty manner. That's on the one hand. On the other, I posted to panix.questions and the post has not shown up yet.
Panix is back with me, or I am back with them. Still no access to CERT.
One article I found (and can't find again) said that this was being co-ordinated through anonymous IRC messages aimed at compromised servers, which then distribute the actual worm. If so--and it's plausible--the question comes up: is this being run in real time?
...and what new "security" measures are the Bushies going to rationalize with this?
Oh, re: your trickle and/or lack of email... I don't pretend to really understand these things, but I imagine that during the night, while panix was out of it, mail from people was not being received or acknowledged as being received by panix so the senders' IPs put your pending mail into a queue for a send-retry later. Prolly your email will start coming through as your senders' IPs retry.
Randolph, I believe that article about the IRC connection is at slashdot. Or, at least I read one like it there earlier this a.m.
In other news, I'm right now chatting in AIM with a friend in London who is in turn on AIM with a friend in Hawaii, on the Big Island, and who says there's an earthquake going on there.
Nothing falling over yet. But I do feel very global.
I imagine it's probably nothing more than the first stirrings of the Mother of All Sturms. Und, you know, Drangs.
Hey, I edited that book. I hope the real Hawaii fares better than the one in the novel did.
Now I know why I was getting 'host not found' errors intermittently late last night. Things seem pretty reasonable now -- Google and Slashdot both came up quick -- but then again I use a little ISP that doesn't seem to be on the target list, and things seem to be handled, now.
Absolutely no adequate words for leaving databases public and unpatched.
Did you? My pb copy must've been released before you started (quite appropriately) adding your name as Ed. to the copyright page. I recall reading a post somewhere in which you said you had at first resisted doing that, but then decided to do it.
Anyway, I enjoyed it. Nothing like a good end of the world story to brighten my day. THE FORGE OF GOD is likewise delightfully catastrophic.
And I was reminded of MoS lately when I heard some mention of the hidden/frozen methane deposits in/under the ocean (something in the science news somewhere), and also that poor little island somewhere off NZ or Australia or something that We Hadn't Heard Anything From following some typhoon or other. Hmm... Come to think of it, I guess I never caught what had happened to those poor people. My image at the time was of that atoll (or whatever) in MoS that pretty much got wiped clean.
Randolph sez: "AP article. Good worldwide coverage, with obligatory incorrect remark from gummint representative."
Which incorrect remark are you referring to? Do you mean this bllsht?
``It's not debilitating,'' said Howard Schmidt, President Bush's No. 2 cybersecurity adviser. ``Everybody seems to be getting it under control.''
It certainly was debilitating, even if it's treatable. I was unable to access Panix in any way during at least a seven-hour window; if that's not debilitating, I'm not sure what would qualify. (Panix seems to be responding properly now.)
I still suspect there's mail catching up to me, caught in other crannies of the internet. Even now, I've only received four or five things since midnight.
For what it's worth, Electrolite and Making Light were also inaccessible last night from over here in attbi.com land. At least one site with IP services from AlterNet was still active at that time (around 1 AM Eastern) and I was able to reach some sites in Canada.
All back to normal at 12:30 today.
No problems with either mail or web service at my end, Teresa. I've been on Earthlink for the past couple of years (and access via cable modem at home through attbi) and found them quite satisfactory for my needs. They even stream video, which I'm debating doing (right now I rely on progressive download).
Michael, I'm still not seeing any mail. It's ridiculous how disoriented that makes me feel.
John, I've been watching another Earthlink user's connections go up and down like an oil-well pump all morning. That may or may not be Earthlink's doing.
Graydon, when I was chatting with Erik Olson about ninety minutes ago, he observed that one response to continuing DDOS attacks would be for major providers to drop links to anyone who's bombarding them.
"Rather than filtering?" I said.
"Filtering takes CPU time," he said, and went off to pour concrete in his basement.
Our here on the West Coasts, 3 hours behind most of y'all, I haven't had any problems today. But it does explain why eveything was so dammed slow late last night when It's usually quite fast. I haven't had any trouble with email myself.
MKK
Our speed is variable, and at its slowest we're crawling. At one point I timed out when I was trying to get through to Google.
I really like Panix. It's a great ISP. I just wish Godzilla didn't always head for Tokyo.
We just went offline entirely for a bit, and our normal broadband connection feels like it's getting squeezed through a soda straw.
What You Said, Kevin.
Drums ... drums in the deep.
One thing I have noticed, the only site I can't access is Caregroup at Harvard Medical where I work. I think they're under seige now (Harvard and Beth Israel Deaconess seem to get whacked whenever any virus gets loose. The I Love You virus, for example, wreaked havoc.)
Teresa, I hope you get across the bridge of Khazad-Dum soon....
Thank you guys so much for setting up this thread. Here I was fearing that my IBook was screwed.
I hadn't realized just how much my social interactions are on-line. I am living with a little old phone line connection and last night and this morning have been a bear. Everyone hang in there.
--claire (who is now having to go off and stand in the cold doing a human connection--handing out flyers for her kid's school fundraiser)
I can easily believe that; in a lot of ways it's the right thing to do, as is attempting to ping that port and refusing to establish any connection if you get a packet back.
An impromptu download speed test has me getting about half the bandwidth I would normally expect, so it looks like the affects are still being felt.
My connection to the web is running a little slow, and I may have some email waiting (though some has definitely gotten through). The newsreader at netaxs.com is down, and I don't know whether that's connected to the DDOS attack.
Graydon, while I can see being angry at the people who failed to take precautions, I reserve my rage for those who are mounting the attack.
My system was fine this morning, then ZAP! in mid everything, down. All logged in and everything, network up, unable to ping earthlink. "Aieee!" cried Legolam, "a Ballhog!"
This wasn't as much fun as it could be.
Probably rate limiting rather than actually cutting a link, at least between major backbones. For one thing, that gigantic gulp of capital they took a few years ago ended up in CPU and link capacities that overran actual traffic demands with the non-appearance of the Messiah projected growth in demand.
Based on the way Network Operation Center control infrastructure was being engineered and deployed at Genuity, I'd guess that operational response to the incident was somewhat suboptimal even at relatively well-staffed ISPs. Some links were undoubtedly shut down out of policy, but some were probably increased in bandwidth, and some dropped because the DWDM switch operators weren't talking to the router configuration guys.
Any time you start turning links on and off in an IP network, routing changes start to take up a lot of bandwidth and CPU capacity on internal routers. Operator responses attempting to fix or ameliorate problems can be extremely destabilizing, and that sometimes causes further response by operators which end up preventing routes from settling, even with advanced routing algorithms and protocols. And of course it is only in crises that operations staff find that circuits which were listed as provisioned and tested turn out not to be, so some changes introduced by routing protocols or by operations will simply fail.
It generally doesn't help clear thought and action when management all the way up to the CEO are demanding immediate, active responses.
On another point, your individual service's slowness will depend how many unpatched M$ installations are connected to the same "DSLAM" or on the same clump of bandwidth provisioned out of your dialtone provider's Central Office where your DSLAM hardware connects to your local loop and thus to your home.
Nancy:
Stomach lining is a finite resource. The world has way more malice in it than I have stomach lining, so it's not something I get angry about. One looks for opportunities to knock such people on the head and generally gets on with life.
Failures of craft, though, actual nonfeaseance of voluntarily assumed responsibility, I *do* get angry about, my detachment not being sufficent for the purpose.
Or, by analogy, it's perfectly reasonable to get angry at a bank robber, but I feel that the banker who left the doors and vault unlocked is due some outrage also.
The MOTD Alexis put up a couple of hours ago said DDoS, and he's in a position to know.
I'm very glad to have things back. I had Web connectivity via Andy's DSL line, but no email or Usenet, and it was very disorienting. That lots of Web sites were slow or missing only added to that.
Kevin--yes, that one. One thing that non-professionals tend to underestimate in network outages is their expense. This is especially so for services that bill per-access or by connect time.
Graydon, save some stomach lining for the architects who designed the building as something with almost no walls, and the city planners who don't care to deal with organized traffic jams.
The Internet was designed as a research network, and very little effort has been made to deal with public policy issues in its conversion to a public network (except those of interest to the entertainment industry.) I've been saying for 15 years that we needed better network management. Sure wish I'd been wrong.
Gawd, doing this from links (all that works for me, at the moment) is hard.
A few points.
1) A NoC will definitly cut the wire if you are spewing worms, and the interconnects are dropping 40% or more of the packets on the floor. The UUNet/Verio IC at MAE-Central was so swamped it basically fell over and died -- this, alone, is the single biggest reason the Internet is so wonky today. UUNet was the big loser in this worm, Verio took it hard as well.
2) Throttling's fine, if you have CPU to spare to throttle. When your routers are falling over from the load, you don't have it.
3) Another bug made this much easier to deal with. Since all the people who didn't patch thier SQL servers also didn't patch thier Cisco Routers, the worm traffic tended to drop thier routers, taking them (and thier worm-spew) off the net. Alas, people not running MS-SQL, but running Cisco IOS, also tended to get knocked down.
4) This was a remarkably efficent worm. It had no payload -- all it does is spread. There's no secondary download -- the ~380 bytes of the initial probe is all there is. And, it uses UDP, so it doesn't take much bandwidth. All this combined to make for something that was excellent at saturating networks -- most any SQL server had enough CPU to saturate whatever Internet link they had.
5) Mail, and the net, are still slow. I'm seeing a 1/3rd of the spam I normally see -- which implies I'm seeing a 1/3rd of the mail I normally see.
6) How would "better network management" have fixed this. Are you positing some central exchange that will test to make sure your systems are secure, and if not, not allow you onto the net.
The reason this worm burned so fast is that it was small, quick, didn't have a mistake in distribution code like Nimda and Code Red did, and it relied on the fact that people who install MSSQL don't know what they are doing, thus, they had unpatched SQL servers on the net.
CERT is back with us, and the CERT advisory on the worm is now up. I find in particular the following to be of great concern:
Compromise by the worm indicates that a remote attacker can execute arbitrary code as the local SYSTEM user on the victim system. It may be possible for an attacker to subsequently leverage a local privilege escalation exploit in order to gain Administrator access to the victim system.
One of the things that most worries me about this attack is that it seems to me much more effective and organized than previous worms. It may be that the disasterous effects of the active worm are covering more subtle and subtly destructive attacks; everyone's attention is focused on the worm right now, and it wouldn't take much of a strategist to predict that and take advantage of it.
As for network operations, I think that since this is now a public network, I think we need to start behaving like it's a public network, rather than a research co-operative. That is to say, have laws and regulations to maintain it, just as we do other public network technologies. I'd want to research this, but the following might be worth investigating:
Another possibly more amusing discussion/mild explanation at:
http://www.theinquirer.net/?article=7418
Rather than government controls, use the market. Force manufactures of shoddy software to take liability for thier products. Force people who put shoddy software on the public networks to be liable for what that software does to other parts of the network. The problem will go away. Companies will no longer tolerate such thigns when it's costing them real cash money, and real bad press. Insurance will help here, as well. 'You are using that? Sorry, we're not willing to write you liability policy.'
As to PGP? Lovely thought. Instead of being jailed for the contents of your mail, you'll be jailed for refusing to divulge your keys and passphrases. That is, of course, if you are lucky. There are nastier ways for someone to get your PGP keys.
It's not like the security mechanisms to defeat this sort of attack don't already exist; they do. Multiple ways of coping with this sort of thing *alreayd exist*.
That they're not implemented is evidence of incompetence, and I know of no case where fixing incompetence through regulation was successful.
Erik, who is there to "force manufactures of shoddy software to take liability for thier products" if not the government? If you say private arbitration I will snicker muchly.
To which I will add that I believe that even the best products will sometimes need to be updated to close security holes, and your liability solution does not address that.
Graydon, that's nonsense--the old FCC did a perfectly reasonable job. A simple fine assessed on violators, with a provision that a net connection be cut off in 30 days if the problems are not addressed, would get the businesses involved to hire competent people or be off the net.
Well, I hope the worst effects of this worm are over, though I see I am still being hit by a UDP probe on 1434 about every half hour or so. Lots of folks apparently let their SQL boxen hang out on the net rather than behind an adequate multi-tiered firewall solution.
As for Erik's proposal, his liability solution also mentioned going after end-users who fail to patch their systems or secure them in other ways. That's exactly the case here as either a Microsoft hotfix or SQL Service Pack 3 would have eliminated this issue and they've been out a while.
One other thing, Steve Gibson certainly is a lively writer and deserves respectful admiration for programming only in assembly but as a security expert he is somewhat lacking. For example, he has been promising to develop security software for years now and he lost the plot over the fact that Microsoft enabled raw sockets support in Windows XP Home.
I was wondering what the problem was last night. This morning, I got a call from my boss at Bank of America about this. Fortunately, our department (credit card applications) is safe (other than the slow network). We use Unix and Oracle on our servers. Other parts of the bank weren't so lucky. I suspect the ATM system was affected by the WAN overload, rather than directly -- the bank accounts are kept on big IBM mainframes. Still, it doesn't look good to the customers. :-(
Just this week I told my boss I didn't want to get a SQL Server that another programmer wanted; I'd rather stick with Oracle because I don't think SQL Server is secure enough. I think this pretty much ends the discussion.
My firewall at home got its first probe at 10:31pm MST, and has had 246 attempts. The last was at 7:46pm, so it's slowing down.
As of bedtime last night, fourteen letters (three of them spam) had managed to squeak through to me. I tried to say so here, but my attempt to post the remark timed out, so I gave it up and went to sleep. Things are better this morning.
I continue to be irritated by Howard Schmidt's remark about the DDOS attacks not being debilitating. Like epidemic viral infections and rolling power blackouts, these attacks cause an incalculable amount of loss, waste, confusion, inconvenience, and miscellaneous distress. Power outages do far more damage, of course, but the principle is the same.
The difficulties are different in every case, but they're nonetheless real. A problem is no less serious for being impossible to summarize in brief.
Friday morning we were about to go off to visit Edinburgh area friends and I made a small pass at Making Light and couldn't get on. Ditto Electrolite. Came back Sunday (this was a no email visit) to find I'd missed all the fun. Explains why I had so few emails, though weekends are usually lighter for me anyway. And this IS Newbery/Caldecott weekend which affects all children's book folk, so I was expecting little.
Hope all is well in webland soon.
Jane
Just for fun reading, a thought experiment: Curious Yellow, a science-fictional (although largely plausible) coordinated superworm.
I suspect it's all those vowels with no place to go. They're piling up, blocking the pipes.
Teresa, you got fourteen letters. Were they vowels?
Okay, that's all I got here. Move along.
Several points, in no particular order...
Point 1: we're already seeing the claims that "if those nasty full disclosure people hadn't talked about it, and had kept it a secret with Microsoft, this would never have happened".
This is, of course, false, on several grounds:
(a) whenever secrecy has been attempted, the information has always got out anyway;
(b) Microsoft's history shows that it will not attempt to fix problems until after they have been widely discussed. (This is generally true of commercial software vendors, in fact, all the way back to the THERAC-25 - google for it, it's a frightening story.)
Point 2: the ARPANET, and the Internet, were designed to be able to cope with much worse problems than this. However, commercial pressures have meant that capacity added to the system has no redundancy and no backup routeing - after all, if you have two 4MBit lines, you can make more money by carrying 8MBit of data rather than having a backup for when one of them gets cut. The redundancy is effectively gone - which makes it very disappointing to see people using the Internet for critical systems as though the redundancy were still there.
Point 3: several models of router have shown a distressing tendency to give up all routeing under this load and require physical resets. There is no excuse for this; it's just cost-cutting, using cheap hardware and badly-written software. This has actually been more of a problem than the worm itself; ISPs with good routers have mostly managed to keep things running even during the initial spread stage - even those ISPs hosting customers with MS-SQL servers. If the internal network of your ISP went down - if you can read a traceroute, you'll know how to check this - you might want to consider asking pointed questions.
Point 4: The MSDE (Microsoft Desktop Engine) is also vulnerable to the worm, and is rather more widely deployed than MS-SQL Server 2000. It gets installed with a variety of other software. Just because you haven't bought MS-SQL Server, that doesn't mean you're not running its buggy code.
Point 5: the fix for the vulnerability used by this worm has been available for six months. Any administrator who has not deployed the patch for it is incompetent - one of the key duties of a system administrator is to keep systems secure. Any company with unpatched and unfirewalled servers is failing in its duty to its shareholders. Class-action suit, anyone?
At the risk of having my ears chewed off by people who know much better than me, I'd like to ask a question or two (with all due respect):
Is this worm also getting into OS X and Unix servers?
If not, does anyone believe this will encourage IS masters to consider switching from Microsoft to a different system for networking?
Reason I bring this up is, this is the second time in two months my hospital has been knocked out for more than a day. This is an expensive outage for an institution that is already way in the red and not in need of more bad press.
My department just got an OS X server, mainly to house and stream a lot of media we need to use with an educational application. Being completely ignorant about the Server world, I can't help wondering nevertheless whether many outfits (mine included) wouldn't be better off running their other networks from a less targetble OS as well.
I ask this, by the way, not withstanding Roger's (to me) excellent Point 5 above. And BTW adding that I author content on both Mac and Win platforms (so no one thinks I'm a complete Apple zombie).
OS X _is_ a unix, so it's not really the fish and tangerines question it would have been a few years ago.
My own take on it is thus:
Yes, the various unix flavours allow better security, BUT:
a)security is a process
b)no obstacle not covered by fire is more than a temporary annoyance.
It all depends on having competent people keeping things secure; it's possible to secure MS systems (more work, sometimes much more work, but possible) but nothing but nothing will save you from incompetent network admins.
As may be apparent, I work in computer security among other things. There are several reasons for a sysadmin not to have patched a problem:
[1] The problem isn't generally known.
[2] There is no patch available
[3] The sysadmin is unaware of the existence of the patch
[4] The sysadmin has not had time to apply the patch
[5] The sysadmin is unable to apply the patch for reasons of system compatibility
In case [1], fair enough - nothing he can do. Case [2] is tricky; one of the reasons I favour open-source is that patches are generally made available extremely fast, But in this case, the patch was available. There's no excuse for case [3]; all reasonably current operating systems run security alert lists to let people know when a patch is available. Case [4] might be because of an excessive load on the admin from management; or it might be because the patch application process is a very lengthy thing (as I gather it was in this case); when combined with an anodyne security alert that says effectively "yes, there's a tiny little problem which you can patch, but it's really minor and hard to exploit", the admin may well decide he has better things to do - especially if it's the tenth patch this month.
Case [5] is pretty rare in a technical sense, though I'm running some older Apache versions on my work servers because of a compatibility problem with other software (and we're not hosting anything of the sort which would be vulnerable to a cross-site scripting attack, which is what those patches fix). Legal compatibility is another matter; Microsoft in particular have tended to roll changes to the licence under which their software may be used into major security fixes, so that you either go on running an insecure server or (as with Windows XP) give them rights to do anything they like to your computer at any time. Thanks, but one day I might want to be able to assert copyright on that manuscript I'm writing...
So, mostly I agree with Graydon; it's usually possible to keep a Microsoft system secure, though certainly not always - and the design of the system (things like using Internet Explorer to render HTML email) sometimes makes it impossible. On the other hand, maintaining security for my Debian Linux boxes is simply a matter of running "apt-get update && apt-get -u upgrade" when a security announcement comes out (and other open-source operating systems tend to have a similar system available).
The headline story is "it's a Microsoft worm". That may well cause a few bosses to decide to switch, but IMHO that's for the wrong reasons - understanding why Microsoft systems tend to be insecure (both because of how MS writes them and because of how they have to be administered) will lead one to the same conclusion but with a bit more reasoning behind it.
The basic problem is that the Internet was designed for researchers to exchange data, and has as its basic assumption the idea that everyone is honest.
To lessen security concerns, a new Internet needs to be designed from the bottom up with the basic assumtion that every man's iniquitous and every man's a liar.
James: yes, absolutely! One of the core problems is that the internet uses in-band signalling - like the old phone systems that responded to blueboxes, it's possible to alter the parts of one's data packet that say where it came from and where it's going independently of the truth. A system which reliably authenticated those data would be... well, it would be IPSEC actually, which has been around for a while and is a Great Good Thing (it has other benefits too). I'd love to see the net switch to an IPSEC-only structure... it's not a universal cure, but it certainly fixes a lot of problems.
However, it also makes it impossible for anyone to eavesdrop on your net traffic.
Graydon, "No obstacle not covered by fire is more than a temporary annoyance" sounds like it belongs in Murphy's Laws of Combat, but I've never seen it in any of the extant compilations. Are you committing aphorism again?
Jim, Roger, et al.: It's surely a problem that the Internet has neither discretion nor prudence built into it; but I have to wonder how much good a more secure set of protocols would do. Users have a tendency to go with unstable and insecure setups, as long as they work most of the time, and allow them to connect in ways they couldn't before.
There you'll be, riveting together I-beams to make a really solid structure, and you'll look up to find the users have half-assedly patched together bamboo scaffolding half a mile past where the poured foundations end, and are clambering all over this horrible improvised structure, doing business and having parties and setting up housekeeping. It's awful. But what can you do? It's where they are, and if you're not willing to watch the structure collapse, or individual users fall into the void through holes in their scrap-plywood flooring, you're going to have to go help them reinforce it and shore it up.
I've been meddling with computers off and on since the days of punchcards, and I've seen a lot of secure and/or proprietary systems come and go. The insecure open-system stuff just sort of swarms over and around them. People don't go where the systems are secure. They go where other people are hanging out and things are happening.
Moreover, as Bruce Schneier and other very smart people keep pointing out, "security" isn't a magic ingredient or a single design factor, so fantasizing about super-duper security architectures that aren't going to happen is a bit off the point. In the last analysis, security is a set of best practices in real time, and nothing beats sustained human attention.
Which isn't to say a high-quality safe isn't more secure than a brown paper bag. Better architectures matter. But as Bruce points out, high-quality safes are rated in terms of time: for instance, a three-hour safe is one that a professional safecracker, with a full set of tools, would take three hours to break into. In other words, even the best material-and-design-based security is nothing more than a delaying tactic. Human attention rules.
Bruce, again: Security is a process, not a product. This worm can be blamed purely on incompetence. The server software was incompetently built (allowing this remote attack) when ship, it was incompetently installed (allowing the internet-as-a-whole to reach the server unfiltered) and incompetently managed (not kept patched.) Seems to me the only competent thing involved was the person who wrote the worm!
No matter how well you design something, if you don't use it properly, it will fail. The world's best locks are useless if improperly installed. I don't care how "secure" your product, protocol or methodolgy is. If you don't competently wield it, it will fail.
This is why $100 deadbolts at home are often useless. The thief merely kicks the door, and shatters the ($5) door frame.
I worry about that sometimes. Short of reengineering the doorway myself -- no use asking my landlord, who won't fork out to repair a malfunctioning bathtub overflow valve -- the most effective strategy would be to put a sign on my door explaining that the second- and third-floor apartments have hollow-core doors and those locks you can pop with a credit card.
But that would be wrong.
There has been an additional complication regarding this worm: It seems that it was possible for an administrator to patch the system, and then (through one of a several possible chains of events) for the system to become unpatched without changing the system registry back.
Unfortunately, Microsoft's software update notification facility only checks the system's registry, not the checksums of the actual application files, so the administrator would not be notified that they needed to re-apply the patch.
More incompetence on Microsoft's part.
The way Bruce Schneier suggests the security problem be tackled is through a combination of manufacturer liability and something like the Underwriter Laboratories (who do the ratings for safes), financed by the insurance industry (as UL is). If you don't use certified equipment, software, and procedures; your information security, loss of business, and other insurance rates goes up.
Teresa - I happened to see a CNN Headline News report on the worm last night. It listed all of the major effects of it, i.e. South Korea getting locked out, routers and e-mail failing, and ATMs being down. Then at the bottom, it had the temerity to list "no damage" as a result. It was a very telling example of media cluelessness.
Teresa:
That's not my aphorism, that's taught to Canadian Forces junior officers as a law of nature. (And I suspect to a great many other kinds of junior officers.)
It fits in with the 'three hour safe' concept; nothing but nothing will stop the other fellows except being defeated; passive defenses won't do more than consume (relatively minor) amounts of time.
(The other aphorism that goes with it is that the ideal for an attack is first shot overkill from outside the other fellow's volume of awareness, though I am fairly sure I remember a non-standard phrasing for that one.)
Better Protocols:
The fundamental problem with these is that they involve putting control into the routers, rather than the nodes; everyone who isn't a really big company can fairly easily figure out what's wrong with that proposal, especially as it doesn't get you anything that tunnels don't, and hardly anyone will take the convience hit to use those.
And, really, this critter is not a fault of the design of the internet; it's pretty purely the result of a collection of specific incompetences. Nothing in the way of highway design will help much if people in cement trucks are driving the wrong way at a hundred miles an hour with their lights off, either.
Andy's remark that "Then at the bottom, it had the temerity to list "no damage" as a result. It was a very telling example of media cluelessness" hit me in the solar plexus.
As someone who is about (Wednesday) to climb into a series of Continental airplanes to go back to the US, the understated fact (CNN) that this affected airports around the world and especially Continental's hub in Newark certainly gave me pause.
Jane
TNH: That's very much why I run WEP on my airport. No, WEP isn't real security, it can be cracked with a not-quite-but-close-to trivial amount of effort. However, the vast number of WiFi points *aren't* defended at all. By merely putting the digitial equivalent of a hook-and-eye fastener, the bad guy would simply drive another half block and get on the wireless hub known as "Linksys."
And, you best bet for a door is A) better hinges and, B) throw bolts that go up into the door header and down into the floor. Then again, the bad guy could just break a window.
Michael: Yes, Windows Update is dumb. However, there is HFNetCheck and the MBSA, which does a much better job of checking for such, and covers SQL, Office the like. Simple Test of NT Admin competence -- do they know of, and use, HFNetCheck? No, it's nowhere near perfect, but it's *far* better than Windows Update, which tells you little about if you are current or not. Info at (amazingly weird URL which, if you fix, just forwards back to the weird one) http://www.microsoft.com/technet/treeview/default.asp?url=/technet/security/tools/Tools/mbsahome.asp
An aphorism that was showing up in fortune files when I was first working full-time with computers: -"If architects built buildings the way software engineers build programs, the first woodpecker to come along would destroy civilization"-.
This was old 20+ years ago, and at the user-functionality level has probably become much less true; the need for programs that can be sold over the counter (instead of as in package of continuing services) has helped, as processors that have sped up enough that careful programming doesn't cost too much user time. But Jim's point about the difference between the design of the net and its current [potential for] use is dead-on; I worked at a company on the ARPAnet, which was designed so groups on the net could use each others' computers, which worked because access to the computers themselves was so limited.
It's a bit like Bester's description in The Stars My Destination of all the new crimes (or variations) that appeared once people learned how to teleport. I've been thinking about The Shockwave Rider this weekend and trying to remember whether any SF writer saw how much simple vandalism the net would enable. How does Neuromancer read when (as in the Steve Gibson link Teresa provides) a thirteen-year-old punk can lob a grenade into the works? It's not as stylish as stealing information, but it can seem similarly destructive.
On a more grounded note: this thread has had a lot of comments about nonfeasant sysadmins. But Steve Gibson's discussion points to subversion of home systems, which just aren't going to be vigorously updated (any more than most drivers will learn how to do oil changes) and have a large potential for causing trouble with the spread of cable modems. I get the impression that most of the commenters don't favor giving Microsoft easy access to all home systems; what practical solutions are there to the problems caused by lots of high-speed connections to/from unsecure systems?
Chip writes: "(W)hat practical solutions are there to the problems caused by lots of high-speed connections to/from unsecure systems?"
Alas, none. The "practical" solution is to make operating systems that default to secure, aren't riddled with holes, and so forth. Scare quotes, because the chances of such being marketed and used are basically nil.
Personally, I'd sharply limit what could go into/out of "basic" DSL/Cable packages (and offer better connections, for more money, to those who know to ask for them. And, with that better connection, comes a more demanding AUP, that says "Hey, we're letting all these ports through to you, you have to secure them. Or we're cutting you off."
And, to repeate Bruce S. (again) -- liability law is probably the only real answer, given the current setup. Software manufacuters have been able, so far, to completly disclaim liability for selling flawed products. When that stops, software *will* become more secure, until that stops, it won't. There's little money in secure, and lots in new features.
I used to suggest some sort of liability for bad programs, and was met with so much horror from programmers that I concluded the industry was so incapable of making programs which could do something which resembled the claims on the box that liability would destroy the industry to no great gain.
Now, I was probably suggesting a more general sort of liability than is being discussed in this thread, and I may have been misterpreting the reactions was getting, but what sort of liability do you have in mind? I suspect that if companies that produce programs with possible security holes are liable for potential damage, then we'll see the same outcome we're getting with medical malpractice insurance. To a very large extent, people can do more damage than they can pay for, and if you try to make them pay for it, they'll go into a safer line of work.
So far as existing law goes, does anyone have any idea of what the liability situation is for companies who sell locks and safes and hire out guards?
The fix for insecure home computers on broadband connections is already in; their service providers won't allow them any open incoming ports at the service provider's firewall, and they'll (often) turn their connection off if they start spewing.
The hard one is infrastructure; a lot of the problem with this worm stemmed from broken backbone routers.
Chip's comment about The Stars My Destination reminded me of Niven's flash crowds. I wonder if, centuries from now, sloppy researchers who've lost track of what order the 20th century happened in will conclude that teleport fiction was written as a metaphor for the Internet.
Erik,
If you think most of these affected systems even *had* someone you could identify as an Administrator (as opposed to someone for who m that was *one* of their duties), you're mistaken. One of the advantages of *nix systems is the fact that a single administrator can manage many more systems, and that remote manageemnt is relatively easy. This changes the economics of employing a systems administration specialist, even part-time. Microsoft's software pretty much insissts on a warm body sitting in front of the box (yes there are solutions to this, but they typically get very expensive).
Nancy, the sort of liability that Bruce S. is talking about is basically criminal negligence. "Did you use Generally Accepted Coding Practises to try and identify all bugs in your system?" Furthermore, "Did you follow industry standard disclosure procedures once you were informed of this flaw?". That sort of thing.
Car manufacturers aren't liable for people killing each other with their products, only for knowingly producing a product which is not as safe as it should be, and/or concealing this fact from the purchasing public.
On the other hand, just as leaving your car at the top of a hill with the parking break off leaves you open to charges of involuntary vehicular homicide. People should be made to take responsibility for the systems they operate. That isn't to say that people are necesarily *liable* for whatever their computer (or vehicle) does, but they are responsible, and are liable if they do not excersize that responsibility to the best of their ability.
It seems worth pointing out that, to continue the architectural metaphor, there is a difference between a building which has a sound structural design and has been poorly constructed or maintained, and one which does not have an adquate structural design. The first will stand, if construction and maintenance are good; the second will fall without extraordinary efforts, and may fall despite all efforts.
Microsoft Windows lacks adequate reliability and security design, whereas most Unixes are fairly strong in this area. I see two main reasons for the problems with Windows: simple uncaring--marketing is a far higher goal than security and reliability to Microsoft--and a design goal of making Microsoft software systems remotely operable by Microsoft and their business connections for various purposes, which leads to numerous backdoors.
This is why various Unix experts are so critical of Microsoft--with MS-Windows and Microsoft's design practices, problems are extremely likely. This worm is just the latest in a long series of problems with MS-Windows. At this time, various Unix and Apple systems are generally superior in security and reliability.
As to the matter of making people who fail to upgrade liable for that failure, there are so many systems involved that the liability to most individual operators is very small and the number of lawsuits that would result is staggering. I think this is one of those situations for which the small fine is made.
Missed the whole thing! A frozen pipe earlier in the week put a 1/4 inch of water into parts of our basement. Luckily, we have a water alarm, so I don't think any actual books were damaged, only the cartons they were in. But about a hundred cartons of books (OK, maybe only 50) had to be moved really fast.
So our basement was in total disarray. I think I must have spent the whole time of the worm attack trying to sculpt piles of cartons into a more pleasing shape, making in possible for children to actually _play_ in our basement -- something Peter has been promised will soon be possible for a number of years.
When cleaning up the piles of books that had lost their cartons to the flooding, I accidentally knocked one off a pile onto the floor. It was a 19th century book with a leather spine and when it hit the floor, both boards fell off. When I pidked it up, I discovered it was a volume from that disbound Poe set we'd beed looking for to send off for hand binding. We'd been looking for the set for about 4 years. (Patrick and Teresa, having had their own adventures exploring the Hartwell basement, are familiar with this kind of experience!)
Anyway, I feel better off for having missed the worm entirely, much improved the situation in our basement, and gound some things we'd lost.
That was well-timed, then, Kathryn. If both events were going to be on the schedule, best to have them at the same time.
Aren't water alarms great? I have a couple I drape over the edge of the tub when I'm drawing a bath (see above, landlord and overflow valve).
Erik, someone who tried to come in through the front window would encounter our real first line of defense: our neighbors. This is not a good block on which to try to do anything unobserved. The time we had a robbery at gunpoint out in the middle of the street was the first time I've ever seen a NYC 911 line choked with calls.
I do worry about that back window. It's the fire escape route, so we're not legally allowed to block it. On the other hand, I could give the appearance of blocking it, which would be very nearly as good.
Chip, Neuromancer's images now seem too elegant and intentional, a sort of hand-tooled boutique approach to breaking into computer systems.
That's as opposed to what Erik was explaining to me yesterday: "If you install Windows NT 4.0 or RedHat 6.2, put it on the net, and attempt to download all the patches, you will almost certainly be compromised before you can get the patches installed. ... There are kiddies out there who run things against whole netblocks."
Gibson wrote about the equivalent of sophisticated second-story cat burglary. What we got was the equivalent of weather.
Nancy, good questions. I'm afraid I don't know their liability status.
Did everyone catch that story yesterday -- Patrick blogged it -- about how one of the corporations whose systems got chomped by this worm was ... wait for it ... Microsoft?
We're about to find out whether Microsoft can be embarrassed. If this fiasco doesn't do it, nothing can.
So can you buy water alarms in your basic hardware store? It looks very much like the paperback shelves are going to have to go in the basement of the new house, which makes me nervous since the prior owner had a dehumidifer down there (and didn't keep anything there that we can tell). We'll get a dehumidifer too, but a water alarm sounds like a great idea.
Bill Gates has no shame. I predict MS will be blaming this fiasco on open source and the open internet before the week is out.
I think it's an exaggeration to say that an over-the-net installation of service packs will automatically compromise Windows NT 4 or 2000 Server before you can get all the patches installed. If you turn on the web server or go URL surfing before you patch, it's a definite possibility. But I've done over-the-net patches a number of times on my single user DSL line to stand-alone computers without ill consequences.
It's also true that I never plug a Windows box into a high-speed Internet connection without installing some kind of hardware or software firewall first -- and then downloading the patches. With W2K, I always leave IIS disabled. RH Linux 8 now has automatic security settings that screen ports in the same way that third-party firewalls screen Windows 9.x or 2000.
I wouldn't say that system compromise of a new machine over the Internet is automatic. If I'm installing on someone else's network, I follow the safer procedure of downloading patches first to an already-locked down machine. Without a firewall pre-installed, you're pretty definitely going to announce your IP address to some people you'd rather didn't know it. And if there are a bunch of other machines connected to the one that surfaces, you're inviting trouble by announcing your presence.
But patching one machine on a single-IP connection (with its ports screened off by a firewall) doesn't seem to me to be as foolhardy as bringing an unpatched LAN server online.
It's not automatic, but you (should) plan as though it is.
Kinda like why you should filter outgoing packets on a per machine basis down to the services authorized and *only* the services authorized in a corporate LAN; it won't stop you from catching anything, but it will slow stuff down when it tries to spread. (and might stymie stuff that wants to phone home.)