Go to Making Light's front page.
Forward to next post: Witness and response
Subscribe (via RSS) to this post's comment thread. (What does this mean? Here's a quick introduction.)
The NOAA-N Prime spacecraft being built at Lockheed Martin in Sunnyvale, CA has suffered an embarrassingly dumb accident:
Earth Science Missions Anomaly Report: GOES/POES Program/POES Project: 6 Sep 2003Check out the photos. I regret to say that the aftermath looks like something out of a Warner Bros. cartoon. (via Tim Kyger)DATE OF ANOMALY: September 6, 2003
LOCATION OF ANOMALY: Lockheed Martin, Sunnyvale CA
DESCRIPTION OF EVENT:
As the NOAA-N Prime spacecraft was being repositioned from vertical to horizontal on the “turn over cart” at approximately 7:15 PDT today, it slipped off the fixture, causing severe damage. (See attached photo). The 18’ long spacecraft was about 3’ off the ground when it fell.
The mishap was caused because 24 bolts were missing from a fixture in the “turn over cart”. Two errors occurred. First, technicians from another satellite program that uses the same type of “turn over cart” removed the 24 bolts from the NOAA cart on September 4 without proper documentation. Second, the NOAA team working today failed to follow the procedure to verify the configuration of the NOAA “turn over cart” since they had used it a few days earlier.
IMPACT ON PROGRAM/PROJECT AND SCHEDULE:
The shock and vibration of the fall undoubtedly caused tremendous damage. Significant rework and retest will be required. NOAA-N Prime is planned for launch in 2008.
Good golly, it looks like a Dalek accident!
Yikes! I shared it with others. Had to.
exiting, shaking my head.
1. Lockheed-Martin is not getting any incentive fees on that....
2. Contractor pays for that bigtime, including for any launch delay costs.... assuming that there are COMPETENT, honest, and dedicated government contract monitors involved, and not FOG whose allegiance is to the contractors being managed and their buddies, rather than the US Government and taxpayers.... Lockheed-Martin literally dropped, the way the contracts as least used to be written, that means that Lockheed-Martin gets to pay the full costs of repair, retest, and program delays. It's not a fun position for a contractor to be in, and it's a very literally unprofitable one, nothing that that sort of thing, to turn what might have been a profitable satellite program, into the red ink column.
3. If there are any competitors left, that's the sort of thing that traditionally helped move government business from the existing contractor, to someone else....
4. Embarrassing things happening in production, are more frequent in the space business than most people might realize. I remember a few from my days in the business. Sometimes the government just rolled its eyeballs if the damage could be fixed quickly and didn't cause delays or raise the costs to the government(thinking of one particular incident the contractor didn;t want the government to officially notice, and which I suppose there was something of tacit agreement on the government side that beating on the contractor would not have been at all productive... sometimes having something on the contractor to use as leverage for other stuff is useful. No, it's not blackmail, not really, especially not when there are things that the government's done that the contractor is willing to be patient on, on the other side. Adversial relationships make for really bad business.....)
Smashed up satellites from inadvertent drop tests on flight hardware, however, goes beyond anything acceptable as overlookable....
I find myself actually blushing in sympathetic embarrassment at the pictures. It reminds me of the time I drove a fork-lift truck into a hearse.
There is something awfully funny about the report, though.
"CORRECTIVE ACTION:
"Lockheed Martin formed an Accident Review Team in which GSFC is participating. The immediate actions concern safety (preventing the spacecraft from rolling, discharging the batteries, and depressurizing the propulsion system). NOAA-N Prime is under guard, all records have been impounded, and the personnel interviewed. After the safety issues are addressed, attention will focus on assessing the damage to NOAA-N Prime."
Innocuous sounding words, note that not one word above denotes "criminal investigation" "sabotage" or "adverse impact and extreme prejudice" but they're connoted.
"The shock and vibration of the fall undoubtedly caused tremendous damage. Significant rework and retest will be required. NOAA-N Prime is planned for launch in 2008."
More understatement. Were the optics mounted yet? Also, anything that depended upon structural integrity and position -- and dealing with a spacecraft that has optics and sensors fed by the optics, alignment is critical and tolerances extremely tight -- basically, that thing's going to have to be torn almost completely apart and everything checked piece part by piece part, and worse. The spacecraft that I worked with, had components that literally had requirements for at least a full year of "burn-in" testing with power on them, before they were considered flight candidate parts to be assembled onto the spacecraft.... anything with burn-in requirements on that bashed bird's going to have to undergo stability testing and characterization all over again.
The good news is that 2008 is four and a half years away -- I don't understand what takes something that's in fairly low orbit, five years of lead time, though, especially since it looked like the bird was fairly far along in assembly and test. It's awfully early to be loading it up with propellants and pressurization.... something seems quite odd to me, about that kind of lead time. I mean, there was a shorter length of time between the time the USA decided to go to Luna and when Saturn boosters started launching!
A cross between a Dalek and C3PO!
I suspect the attention to detail signified by the invitation at the bottom of the page: "Sign up Now for our Free Newsleter".
Ouch!
The mishap was caused because 24 bolts were missing from a fixture in the 93turn over cart94. Two errors occurred. First, technicians from another satellite program that uses the same type of 93turn over cart94 removed the 24 bolts from the NOAA cart on September 4 without proper documentation. Second, the NOAA team working today failed to follow the procedure to verify the configuration of the NOAA 93turn over cart94 since they had used it a few days earlier.
In other words, somebody "borrowed" a couple dozen critical bolts and the crew didn't check them.
Somebody's gonna be in deep, deep trouble over this. I just hope it's the right person/people, rather than a convienent scapegoat. (When I worked for the military, embarrasing mistakes were invariably blamed on either a secretary or a draftsman.)
So, if the other team removed the 24 bolts... why? Was -their- turnover cart missing numerous bolts? And if so... why?
It's pretty clear that the satellite got smashed up because its team failed to doublecheck the turnover cart. Regrettable, but...
...but it's that other team, who cannibalized parts from the satellite team's equipment, that seems to be a symptom of MUCH bigger problems. Is there a bolt shortage? Why? Are other parts in the same situation? Why?
"Really big owie" is right. Think, oh, 50 or 60 top-of-the-line Ferraris simultaneously being dropped from a high place. My stomach hurts just looking at it -- makes me think of the time in college when I managed to pull a rather large laser off an optical table onto the floor, multiplied by a few thousand.
I keep imagining the thoughts going thru the head of the poor guy with his finger on the cart lift button as he watched the satellite sli-i-ide and ti-i-ip. They say the NTSB finds the last word on most recovered cockpit voice recorders is "Shit!"
BTW, POES is a series of polar orbiting weather satellites. You can see the POES program plan here
There are supposed to be 2 flying at a time, and each one lasts 4-5 years. NASA bought the last 4 (POES-L, M, N, and N') all at once; they'll be replaced by a new series (NPOESS) after N'.
N' is basically a spare; L and M have launched, and N is scheduled for next June, 2 years after M, but N' was obviously ordered in case one of the others failed. N' was scheduled to be available to launch any time after late '04. As long as N works, and M lives a normal lifespan, they have plenty of time to fix this one; if N goes in the drink, Lockheed will have to scramble.
The bad news is, it looks like they were ahead of schedule in integrating N'; except for solar panels, it looks like the bird is largely complete. The good news, if any, is that it landed with most of the instruments on top. Using an umpteen million dollar spacecraft bus as a crumple zone is not something one tries to do, but it's better than landing with the instruments on the bottom...
Awe inspiring... I think I need to print one of the pictures out to hang over my desk, to remind me how happy I am that my work mistakes can't cause this much damage. I could probably lose my employer a few hundred thousand if I did something exceptionally stupid - but nothing on this scale.
Very nice to see the comments from people who've worked in the industry.
The first place that I saw this one was as one of the Particles on Electrolite.
Team B probably didn't borrow the 24 missing bolts. Perhaps they started to replace the fixture with a different one but stopped, for some reason. Team A, seeing the original fixture in place, neglected to verify that all the bolts were still inserted.
Sabotage by Team B can't be ruled out.
Jordin Kare writes: ""Really big owie" is right. Think, oh, 50 or 60 top-of-the-line Ferraris simultaneously being dropped from a high place."
Or like that shipload of BMWs, Volvos, and Saabs that went down in the English Channel.
If you look closely, you can see some of the cars, pretty banged up, in this picture of a recovered section of the ship.
Clearly it's not a parallel situation, but was I the only person whose thoughts turned irresistably to Mme Ekaterin Vorsoisson upon first seeing this linked from Particles the other day? :-)
Unfortunate though it is, my mind was irresistably drawn to the Watterson line (and book title)
Scientific Progress Goes "Boink"
>Scientific Progress Goes "Boink"
I prefer the classic line, "Experience is directly proportional to equipment ruined." I'd say Lockheed-Martin can now proudly claim to have some of the most experienced technicians in the industry.
> was I the only person whose thoughts turned
> irresistably to Mme Ekaterin Vorsoisson upon
> first seeing this
Never occured to me until you said it, but my God, yes!
From the description of things, it would almost sound as if these were some bolts that might be reasonable to remove, and that one might almost reasonably not notice were missing in the process of moving things. At least, that's how I read it.
Then I looked at the picture. This satellite is mounted on a six-foot-diameter metal pad, and this pad is mounted on a corresponding pad on the cart, and there are about 36 currently-empty bolt holes that connect the two. (My guess is that some of them weren't used.)
This means that somebody was standing next to this large and undoubtably heavy spacecraft that is much taller than they are, and was removing every single bolt that was holding it up, leaving it some approximation of merely balanced in place. And these are certainly Not Small Bolts.
I cannot honestly imagine that removing these bolts without otherwise securing the satellite was either safe or included in the allowable proceedures, regardless of documenting that it had been done afterwards.
Funny!
If I worked there, I'd have to suppress the giggles. That's probably why I've never been entrusted with anything valuable.
Once, though, I forgot to set the emergency brake on a big delivery truck that I was driving...and while I was placidly eating my lunch, the truck rolled out into the street - smashing itself, an innocent car, and part of the truck's payload of automotive windshields.
I drove the truck back to the office, parked it in the lot, went immediately home and didn't answer my phone for two weeks.
Oh, and another time, when I was working at a plant nursery, I accidentally turned on the overhead sprinkler system when the place had a full load of customers, soaking the lot of them. Yes I did.
So spaceship droppers, take heart. We all make mistakes.
I have to agree with Brooks; if the bolts were indeed 'borrowed' by another team, it looks like something a lot closer to sabotage than an accident.
I'm wondering if instead, it was never bolted in in the first place? Mind you, anyone who lowered the spacecraft into place and didn't bold it in would be negligent, but it seems more likely than someone going to all the effort of unbolting them.
Harriet - [smacks forehead] That's IT! That's why this sounded familiar.
Thanks, that was driving me crazy.
I work for LM. This "accident" came up in several meetings. The general consensus? "Thank God I don't work in that division.".
Two major mistakes: Removing the bolts (big no-no) and failing to check them before moving (big no-no). Both cases, someone willfully disregarded procedure. They'll be lucky to be merely fired.
From what came up in staff, by the way, it was bolted in place before. After attaching (and bolting) the sattellite, they took a break (lunch, overnight, it wasn't clear) and were going to move it "later". Merely attaching the satellite to the truck is a several-hour job.
I have an image of somebody going to their boss - "things would go a lot quicker around here if we had a set of bolts for each cart" and the manager decides to save $100 or so.
Somewhere, there's a Robot Wars robot with some really bitchin' protective metal bolts all over it.
The poor baby was essentially complete, with all of the major instruments loaded. You can see the HIRS, AMSU-A1, MHS and AMSU-A2 in the side view picture. The AVHRR is hidden behind the HIRS. NOAA-N can be seen in the background. Since the AVHRR and HIRS are Cassegrain telescopes, you can guess what happened to the optics.
Note that the factory is in Sunnyvale. Sunnyvale is in California, which has earthquakes. Leaving a $239M spacecraft unsecured in an earthquake zone is beyond irresponsible. They should never have sold the East Winsor New Jersey factory.
The immediate problem is how to move N' so that N can be transported to Vandenberg for launch next September.
BTW... this buy was for KLM, with N and N' added on. It is called N' rather than O because OPQ was intended to be the next block buy until NPOESS came along.