Back to previous post: Hey, hey, it’s the Fourth of July

Go to Making Light's front page.

Forward to next post: Do I dare disturb the universe?

Subscribe (via RSS) to this post's comment thread. (What does this mean? Here's a quick introduction.)

July 6, 2012

Holding vulnerability in the palm of your hand
Posted by Abi Sutherland at 07:46 AM * 69 comments

In the most recent thread on banking technology and how it goes catastrophically wrong, Dave Bell linked to an article in The Register about the possibility of an Android spam botnet. Interesting stuff.

Terry Zink, who blogs about cyber security on the Microsoft Developers’ Network, posted an entry entitled Spam from an Android botnet. He writes that, based on the header and footer information from spam sent from compromised Yahoo! accounts, he suspects that there’s a botnet running on Android mobile devices.

Geo-location of the IP addresses points to phones in Chile, Indonesia, Lebanon, Oman, Philippines, Russia, Saudi Arabia, Thailand, Ukraine, and Venezuela. As Zink says:

I’ve written in the past that Android has the most malware compared to other smartphone platforms, but your odds of downloading and installing a malicious Android app is pretty low if you get it from the Android Marketplace. But if you get it from some guy in a back alley on the Internet, the odds go way up.

I’ve also written that users in the developed world usually have better security practices and fewer malware infections than users in the developing world. Where are almost all of those countries in the list above? Mostly in the developing world.

The Reg article prints Google’s denial, suggesting that the headers and footers have been altered, and that the botnet is really run from PC’s. Zink says he considered that possibility, but finds the Android botnet explanation more plausible.

Investigation continues.

But before those of us with iOS devices get too smug, let’s remember that a walled garden doesn’t so much prevent problems as monopolize the industry of problem provision. Because the other mobile-app story from the last few days is on Apple devices, and comes via Instapaper’s Marco Arment. He released a new version of his app on July 4, and it didn’t go so well.

Last night, within minutes of Apple approving the Instapaper 4.2.3 update, I was deluged by support email and Twitter messages from customers saying that it crashed immediately on launch, even with a clean install.

Arment’s team had tested the app. Apple had tested the app. But the app that people were downloading was crashing on launch. It took a couple of hours (during which time Arment garnered a lot of one-star reviews from his keenest users) for a working binary to be available on Apple’s servers.

And Instapaper wasn’t unique. Arment named 114 apps with the same problem before he quit keeping the list up to date, having proven that it was a pattern. Goodreader was also affected, and also blogged about the issue. Yet another prominent victim was the Angry Birds franchise, so this is visible to the casual app-user community as well as the iOS power users.

According to the Guardian, Apple have acknowledged and fixed the problem today (July 6). They’ve also deleted the one-star reviews.

In a statement, Apple said: “We had a temporary issue that began yesterday with a server that generated DRM code for some apps being downloaded, it affected a small number of users. The issue has been rectified and we don’t expect it to occur again. Users who experienced an issue launching an app caused by this server bug can delete the affected app and re-download it.”

Arment disputes the characterization of “a small number of users”, and that the problem arose on July 5. So there’s still some truth yet to come out. And users with in-app data will be reluctant to delete and reinstall, since they’ll lose their data if they do so. (Apple have apparently reset the update flag on the damaged apps now, so that users can dowload working versions and not lose their data.) (Goodreader’s blog entry explains how to get around the issue with their app.)

Without wanting to be seen to be obsessed about that Economist article, I’d just point out that the much-lauded “pay with your mobile phone” revolution will work a lot better after we get serious and clear-minded about the risks of these computers we’re carrying around and depending on. That includes finding ways to ensure that the software on them is genuinely reliable.

Comments on Holding vulnerability in the palm of your hand:
#1 ::: Victoria ::: (view all by) ::: July 06, 2012, 10:23 AM:

All I can think of at this point is, "If it's easy for you to access your stuff, then it's easy for others access it illegally, too."

I'm not a luddite. I love technology but I know just how easy it is to find workarounds. That's also why my cell phone is just a phone and not a miniature computer and game console.

#2 ::: abi ::: (view all by) ::: July 06, 2012, 10:36 AM:

Victoria @1:

I once had a colleague who did a Unix security course. On the last day, the instructor gave them a fresh new server, blank as a baby's mind, and told them to secure it before they went to lunch.

They removed the keyboard, mouse and network cables, then locked the door and went to lunch.

That's one way to secure a system. I think you can see the flaws in it.

You're using a similar way to secure your data. But there's a huge population of people who need to, or want to, do things with their phone that require a bit more functionality.

In the nicest possible way, I would rather not see this turn into a "thank goodness I don't use these smartphone thingies!" conversation. Not everyone is as graceful and un-smug about their choices as you are.

And it doesn't lead to actual, practical and nuanced discussion of the realities of the situation.

#3 ::: Kevin Riggle ::: (view all by) ::: July 06, 2012, 10:39 AM:

That includes finding ways to ensure that the software on them is genuinely reliable.

I mean, the point of the Apple DRM which failed in this case? Exactly that of ensuring genuinely reliable software. Sometimes processes, any processes, have faults, and misbehave, and it sounds like Apple has mostly done the right things in correction.

Google, for all its faults, also seems to largely do the right things, if the Google Play Store (nee Android Market) is mostly clean, and it's the downside of an open platform that doesn't lock you into a single source for apps that some of those sources will be actively malicious. User education more than some technical solution seems key here.

Tangentially, I have been pleased to discover that smartphone mobile data service has vastly improved at my folks' place in rural Iowa. It turns out (quite seriously) that farmers buy a lot of smartphones.

#4 ::: Andrew Plotkin ::: (view all by) ::: July 06, 2012, 10:49 AM:

"They’ve [Apple has] also deleted the one-star reviews."

So there's a question. These are not trolls or spam reviews. Yet they are also obsolete and do not reflect the quality of the app. I don't like the idea of making them vanish, nor the idea of leaving them in place. What's the collective-review-site equivalent of disemvowelling?

(Apple's site has a distinction between "reviews of this version" and "reviews of previous versions", which is a related idea, although it wasn't applied to this particular case.)

#5 ::: Dave Fried ::: (view all by) ::: July 06, 2012, 10:52 AM:

Regarding Google and Google Play...

When I was on AT&T, I didn't buy a smartphone because they wouldn't let you side-load apps.* Now that I'm on Verizon and have owned an Android phone for several months, I haven't actually ever side-loaded an app. All of my apps come from Google's store.

At some point, I'll probably want to. But in the short term, it's better for my peace of mind to know someone is serving as a gatekeeper for what's on my mobile device.

* From what I understand, they've since changed their policy.

#6 ::: Serge Broom ::: (view all by) ::: July 06, 2012, 10:59 AM:

Should it make me nervous that Google's tablet is called Nexus 7? If I get one, maybe I'll call it Pris.

#7 ::: John Chu ::: (view all by) ::: July 06, 2012, 11:24 AM:

It looks like what Apple did was move every review for the corrupt versions from "Current Version" to "All Versions." That's a middle ground between "making them vanish" and "leaving them in place", I suppose. Only people looking for older reviews will find them.

The non-corrupt versions also trigger an update even though version numbers haven't changed. That means the people who were affected by this issue (not me, for the record) can fix the problem without losing their data.

(I suspect these two actions are not unrelated. No idea if Apple does any comment moderation at all...)

#8 ::: abi ::: (view all by) ::: July 06, 2012, 11:42 AM:

John Chu @7:

Those are both eminently sensible moves on Apple's part. They weren't in place when Arment wrote his blog entries, but then, Apple hasn't yet admitted that there were problems that far back in time.

I don't know if Apple did comment moderation before (or even some kind of ratings filtration and reporting), but I suspect they're considering starting. I would, seeing as how that was the first warning that there was a problem.

Apple's in the same difficult position any service provider is after a problem: how much do you admit fault (and erode trust in the system) and how much do you wait to see who's noticed and shouts (and erodes trust in the system)?

Personally, I think that saying the problem started on the fifth when Arment started talking about it on the fourth is poor strategy. But they're clearly on the right track in other ways.

#9 ::: j h woodyatt ::: (view all by) ::: July 06, 2012, 11:59 AM:

These two incidents are qualitatively very different in a way that illustrates an important principle of systems design: the trade-off between usability and security.

From a usability perspective, the whole point of any security protocol is to diminish usability. At the very least, to diminish it for those without authorization. Often there isn't any good way to do that without also requiring users to authenticate themselves somehow, which can be work, sometimes painstaking work. In other words, when all the doors and windows in the neighborhood have steel bars and triple-deadbolt locks, everybody loses somehow

Likewise, security protocols, like any software, can sometimes be found in error, allowing access by unauthorized users and/or denying access by authorized ones, which additionally impairs usability. The two cases above are different in exactly this way, and a debate could be had about which is worse: allowing unauthorized access or denying authorized access. Still... they're qualitatively different, and shouldn't be confused.

p.s. I think if I were trying to find a supporting example of "Look, iOS has the same problem that Android does" then I'd start by looking here.

#10 ::: abi ::: (view all by) ::: July 06, 2012, 12:21 PM:

I'm not saying that iOS has the same problem that Android has. I'm saying that iOS's solution to the fundamental issue is different, and has its own problems.

They shouldn't be confused. They should be contrasted, considered, and used for parallax.

#11 ::: albatross ::: (view all by) ::: July 06, 2012, 12:24 PM:


Yeah, this distinction is important in a lot of places--every security system has some level of false rejects (you're locked out of your house and have to call a locksmith to get in) and false accepts (I get into your house while you're gone and help myself to the TV, computer, and chocolate chip cookies).

And related to this is the extra work done by both legitimate users and attackers. In the best situations (like crypto), I can make the attacker's job more-or-less impossible at small cost to the user. In worse situations (like password security), I can make the attacker's life twice as hard by making the user's life twice as hard. In still worse situations (like trying to retrofit security into badly-written software), the user does a lot of extra work (or spends a lot of extra money) trying to secure it, and gets very little improvement in security.

#12 ::: Larry ::: (view all by) ::: July 06, 2012, 12:30 PM:

Apple has basically reset the update flag on apps that were affected so no data will be lost, this also has the affect of hiding the reviews for that old, broken version. No need to delete/reinstall apps.

#13 ::: Dan R. ::: (view all by) ::: July 06, 2012, 12:46 PM:

...and [blank] in an hour.

I've racked my brain for an hour and I can't fill in the blank.

#14 ::: abi ::: (view all by) ::: July 06, 2012, 01:09 PM:

Dan R @13:

I'm confused. What are you referring to?

#15 ::: Mary Aileen ::: (view all by) ::: July 06, 2012, 02:01 PM:

Dan R. (13): 'eternity'?

#16 ::: Dan R. ::: (view all by) ::: July 06, 2012, 02:10 PM:

Abi: "Hold infinity in the palm of your hand" is from William Blake's Auguries of Innocence

To see a world in a grain of sand,
And a heaven in a wild flower,
Hold infinity in the palm of your hand,
And eternity in an hour.

I thought you were making a sly quote, and was looking for an equally sly quote in reply.

#17 ::: abi ::: (view all by) ::: July 06, 2012, 02:33 PM:

An unconscious echo, not a conscious one. Must reread and drift it back up the levels of the brain.

#18 ::: David Harmon ::: (view all by) ::: July 06, 2012, 05:24 PM:

albatross #11: It's worth noting that this is fundamental to any security system. Even our own immune system misses some infections (not to mention cancer, representing "hostile insiders"), or occasionally turns on the rest of the body (lupus and other auto-immune diseases).

#19 ::: Bruce Cohen (Speaker to Managers) ::: (view all by) ::: July 06, 2012, 05:33 PM:

albatross @ 11 touches on the tail fluke of the blue whale1 in the room: a large amount of the unreliability in software is also (and much more importantly) a security issue. The culture of software development has been, for the last 4 decades at least, focused on fast and and cheap of the holy trinity "fast, cheap, good: pick two". But the most common kind of simple overrun, off by N, and bad pointer bugs are serious precisely because they're the easiest kinds of problems to exploit as security flaws. And that won't change until the software culture changes its attitude about the severity and importance of these kinds of problems, and the need to eliminate them before software is shipped rather than after.

Of course there also needs to be a complete rethinking of the architecture of computer systems, pushing the concern for security down towards the platform (and not by developing a so-called "trusted system" which is just another way of saying we'll let the OS vendors make all the mistakes). But I don't really believe this will ever happen.

1. Oh, so that's where that huge pile of wet krill came from!

#20 ::: Serge Broom ::: (view all by) ::: July 06, 2012, 06:44 PM:

You can do it fast or you can do it right.

#21 ::: Autarch ::: (view all by) ::: July 06, 2012, 06:58 PM:

The only truly secure system is one that is powered off, cast in a
block of concrete and sealed in a lead-lined room with armed guards --
and even then I have my doubts.
-- Eugene H. Spafford

#22 ::: Autarch ::: (view all by) ::: July 06, 2012, 07:32 PM:

Bruce Cohen@19

On a serious note:
Fast & cheap is one issue. Sometimes the issue with the culture that is widely shared. E.g. security through obscurity. Another is the attack that was not taken into account. (E.g. port 25 not originally being designed to be more secure, not sanitising database inputs1)

Or maybe this is a form of the cheap+fast options.

1 Calling Robert Tables to the courtesy phone.

#23 ::: Jim Macdonald ::: (view all by) ::: July 06, 2012, 10:28 PM:

The problem with any tech is that it has risks.

Automobiles gave us modern American culture. They also killed 50,000 people a year.

We haven't yet figured out seatbelts and airbags for computers.

#24 ::: mjfgates ::: (view all by) ::: July 07, 2012, 02:10 AM:

We are figuring out general ways to deal with security issues. As time goes by, a problem that used to require genius-level thinking to avoid fades into something that can be dealt with by following normal best practices, and eventually into something that you have to work extra-hard to cause. Bad pointers, for example, mostly can't happen on either Android or current Windows systems; those operating systems are written in managed languages that do not have pointers. Handling the "Bobby Tables" problem doesn't take Einstein with regular expressions anymore-- just use parameterized queries-- and as data-manipulation mini-languages become more closely integrated into primary programming languages, it will go away completely.

At the same time, the people who want these systems broken will go and hunt down new classes of problem. I don't see an end to this particular arms race; the analogy to parasite and host is disturbingly good.

"Fast/cheap/good" is a red herring. Remember those occasional space probes that blew up because one programmer used feet where another one was expecting meters? That's software that went as far out toward "good" as possible-- every line looked at six different ways-- but you still get bugs. Trying harder to avoid bugs stops working after a while, or worse yet, starts causing them.

#25 ::: Charlie Stross ::: (view all by) ::: July 07, 2012, 05:53 AM:

More recently: a piece of malware made it into the iOS app store, passing the app approval process.

(Details here. TL:DR; it's Russian malware, uploads your address book and GPS coords to the author's server, then spams your contacts with SMS spam, starting with an invite to download the app.)

Apple's response is instructive: they yanked the app within single-digit hours of Kaspersky AV reporting it to them.

It is unclear whether they used the "remote kill" facility of their DRM server to nuke installed copies of "Find and Call". Given the DRM server corruption that occurred a day earlier, it's unclear whether they could have done so.

I find the temporal proximity of "DRM server go la-la" and "first piece of iOS malware identified" highly suggestive, though. (Did some bad guys execute an attack on Apple's store infrastructure to try and cover roll-out of an iOS botnet by hampering Apple's ability to police their walled garden? Or did someone at Apple screw the pooch while trying to take down a botnet-in-the-making?)

#26 ::: Charlie Stross ::: (view all by) ::: July 07, 2012, 05:54 AM:

Oh noes! The gnomes think I'm an imposter!

#27 ::: David Harmon ::: (view all by) ::: July 07, 2012, 07:26 AM:

mjfgates #24: Some of NASA's problems can be traced to institutional problems -- where for external or Iron Law reasons, the managers actively interfered with the engineers. (*cough* O-rings *cough*)

#28 ::: Larry ::: (view all by) ::: July 07, 2012, 06:53 PM:

Bruce Cohen@19: The sad thing is there are a number of tools in place to help catch these things. Things like valgrind, purify and other tools really do help reduce problems. But the bigger issue is developer mindset. They don't program with things like security in mind, and many don't handle errors very well.

I don't think the architecture needs rethinking so much as the way people program.

#29 ::: Serge Broom ::: (view all by) ::: July 07, 2012, 07:04 PM:

David Harmon @ 27... (*cough* O-rings *cough*)

"For a successful technology, reality must take precedence over public relations, for nature cannot be fooled."

#30 ::: Andrew Plotkin ::: (view all by) ::: July 07, 2012, 11:04 PM:

Charlie@25: '"I find the temporal proximity of "DRM server go la-la" and "first piece of iOS malware identified" highly suggestive, though.'

Slightly suggestive, maybe. There are an awful lot of hidden causal links you have to postulate to connect the events up.

(Why do we believe this is the *first* piece of iOS malware? What does that mean? This is not the first app Apple has removed from the store, nor the first app to improperly upload Address Book data, etc.)

#31 ::: Andrew Plotkin ::: (view all by) ::: July 07, 2012, 11:28 PM:

Larry@28: "But the bigger issue is developer mindset."

Yes and sort of. (Various strained analogies were attempted here and then deleted from the post.)

Programming will always be difficult enough that the programmer can't pay attention to every detail. This is axiomatic, because whenever you invent more powerful programming tools, we use them to *tackle more complicated problems*. Everybody is always out on the bleeding edge of their abilities.

I like to think I am a skilled and careful programmer -- by which I mean, I write code one degree *less* complicated than the most complicated code I could write. This means that I can stay on top of things; when something gets unexpectedly nasty, I don't lose control.(*)

Nonetheless, I make mistakes. Yes, I use the prophylatic error-checking tools available to me. That eliminates some of the mistakes, leaving the rest of the mistakes undetected.

(* Mostly. Let's not talk about the text-scrolling code I was wrestling with this afternoon. Please.)

So, Bruce mentioned off-by-N and bad pointer bugs. These are particularly awful because large chunks of the programming industry work in C-level languages. In other words, we're using languages where nearly any mistake is a potential security problem. This is clearly idiot-level planning (except of course that nobody planned this situation, it "just happened"). Telling people to adopt a mistake-free mindset is not going to help, though. You get people to adopt tools where mistakes don't *do* that.

To their credit, this is what Google tried to do with Android. They picked a security-conscious virtual machine to run their apps on. Turns out, unfortunately, that no technological feature of Java is proof against Sun being bought out by Oracle, hammered flat, and stamped into patent ammunition.

#32 ::: Doug Burbidge ::: (view all by) ::: July 08, 2012, 12:21 AM:

To see a silicon chip in a grain of sand,
With software to empower,
Hold vulnerability in the palm of your hand,
And an exploit in an hour.

#33 ::: clew ::: (view all by) ::: July 08, 2012, 01:40 AM:

"I am a skilled and careful programmer -- by which I mean, I write code one degree *less* complicated than the most complicated code I could write."

This was one of the most practical pieces of advice in William Kahan's Numerical class (actually, History of Computing, hardware emphasis, and How it Should have been Done). Code on your third-best days, test on your second-best, hope you have enough first-best to debug.

#34 ::: Bruce Cohen (Speaker to Managers) ::: (view all by) ::: July 08, 2012, 07:55 AM:

clew @ 33:

I can also use Kahan as a good example of a bunch of architectural decisions that removed security and reliability in favor of premature optimization. Kahan's original work on the IEEE floating point standards included support for detecting and correcting overflow and underflow errors, and for interval arithmetic. In fact the implementation of the first couple of floating point coprocessors at Intel, whose algorithms and code were designed by a student of Kahan's, contained all that support. Then designers at AMD came out with a processor that dropped part of that support1, and later Intel processors did as well, because the hardware engineers weren't willing to dedicate the required level of on-chip resources (gates, registers, interrupt levels, etc.) for that functionality. And then subsequent versions of the IEEE standard made those functions optional, and no one was willing to even put them in the software floating point packages.

The result was that a good part of the time even the error detection features were turned off by software because no one wanted to deal with the complexity of handling the errors reasonably.

1. I was responsible for writing the evaluation tests for the first systems that used the 8087 floating point coprocessor, and for the ones that used the Intel second-source parts for the AMD 9511 and 95122. I discovered those and other short-cuts in the 9511 design when I tried to run the same set of tests on both the 8087 and the second-source of the 9512.

2. How Intel came to second-source those and a few other AMD parts is a fascinating tale of (rumored) industrial espionage and corporate cooperation-under-duress, that I heard about only because I had worked at AMD before being hired at Intel. The story goes that an AMD chip designer was hired by Intel and brought with him engineering documents (schematics and perhaps tapeout diagrams) of several AMD parts and that when AMD discovered this they threatened prosecution unless Intel allowed them a second-source agreement on some products in addition to the 8086 second-source that Intel was offering (under pressure from IBM, who wanted to ensure supply of parts for the IBM-PC).

#35 ::: Bruce Cohen (Speaker to Managers) ::: (view all by) ::: July 08, 2012, 08:50 AM:

Andrew Plotkin @ 31:

You get people to adopt tools where mistakes don't *do* that.

Yes, that. And you create interfaces and protocols that include parameter checking, and you bloody well handle major errors like divide-by-zero. Where possible and suitable you have default error-handling boilerplate automatically generated by your IDE, so it doesn't get forgotten.

As for redesigning architecture, read up on "capability-based access" and the Intel iAPX 432 to see what we might have had. And an exercise for the reader: Why was Linus Torvalds wrong for considering the microkernel OS architecture a bad idea?

#36 ::: Michael I ::: (view all by) ::: July 08, 2012, 09:22 AM:

David Harmon@27, Serge Broom@29

Although another important point (at least from Edward Tufte's account in the "Visual and Statistical Thinking..." chapter of Visual Explanations ) is that the engineers' presentation did not present a convincing case.

"As analytical graphics, the displays failed to reveal a risk that was in fact present. As presentation graphics, the displays failed to convince government officials that a cold-weather launch might be dangerous. In designing those displays, the chartmakers didn't quite know what they were doing, and they were doing a lot of it."

(The "displays" being the charts prepared by the engineers in support of their recommendation to postpone the launch.)

Now I'd actually suggest that simply having your engineers say that they think a launch under the expected conditions is too risky is by itself sufficient grounds to postpone a launch, at least unless one has clear evidence that the engineers are mistaken. But, even so, the argument for postponement certainly becomes stronger if a connection is clearly demonstrated between the expected conditions and likely risk.

#37 ::: Serge Broom ::: (view all by) ::: July 08, 2012, 09:56 AM:

Michael I @ 36... I wasn't aware of attempts made before the launch to say "This is not a good idea", but I seem to remember an interview with Feynman after he made his famous demonstration. Basically, one of his friends, who had talked to the engineers, approached him and made some comments that shepherded him toward the cause without having to outright tell him because the engineers were afraid for their jobs. Feynman appeared amused at having been led on. (BTW... His demonstration can be found on YouTube.)

#38 ::: abi ::: (view all by) ::: July 08, 2012, 10:04 AM:

I've heard it suggested that you should write your software as if it's going to be maintained by a short-tempered psychotic with your home address.

There's a certain value in that.

#39 ::: Michael I ::: (view all by) ::: July 08, 2012, 11:12 AM:

Serge Broom@37

According to Tufte's account, the basic sequence of events leading up to the launch was:

1) The day before the launch, engineers at Thiokol (the rocket maker) recommend postponing the launch. Thiokol managers concur in this recommendation. The engineers fax 13 charts to NASA in support of the recommendation to postpone.

2) The charts are discussed in two teleconferences between NASA and Thiokol. NASA officials point out weaknesses in the charts and argue for reconsideration of the recommendation to postpone.

3) Thiokol managers change their minds and oppose postponement. The decision to launch is made.

#40 ::: Serge Broom ::: (view all by) ::: July 08, 2012, 11:43 AM:

Michael I @ 39... I wonder if those mgmt bozos later had their asses handed to them.

#41 ::: Serge Broom ::: (view all by) ::: July 08, 2012, 11:44 AM:

abi @ 38... That'd work. My approach to programming is Murphy's real law - if someone can do it wrong, someone will.

#42 ::: Michael I ::: (view all by) ::: July 08, 2012, 12:08 PM:

Serge Broom@40

Don't know what happened to the managers.

And I'm not suggesting that the decision they made was the correct one even given how they understood the evidence. Tufte's point is that the engineers' presentation didn't make a convincing connection between cold weather and increased risk. So we're left with the more difficult question of whether the expressed beliefs of the engineers in the absence of convincingly presented supporting evidence should be enough to postpone an unusually high-profile launch.

(I'd tend to say it should be enough. But I'm not managing a project that's in line for a possible teleconference during a State of the Union address.)

#43 ::: Serge Broom ::: (view all by) ::: July 08, 2012, 12:20 PM:

Michael I @ 42... Good point. And hindsight *is* a wonderful thing. (Although that can be used as a convenient excuse by people who should have listened to the experts with the assumption that the latter might know more about the area of expertise than the decision-makers.)

#44 ::: Dan R. ::: (view all by) ::: July 09, 2012, 08:57 AM:

Doug @32: I am your student.

copied somewhere off the web years ago:

the sand remembers
once there was beach and sunshine
but chip is warm too

#45 ::: Henry Troup ::: (view all by) ::: July 09, 2012, 10:22 PM:

The issues with software development, IMO, are nearly all people problems. People demand a lot of function in too short a time. People cut the budget so that test tools can't be purchased (or, won't buy enough licenses to actually use the tools) or they won't trust developers with security assessment tools. People insist that the new technologies are too risky, or that ancient, crufty, and broken clients (I mean IE 7) must be supported, long after much of the industry has dropped them. People interpret specifications in ways the writer never intended. People look at their work and say "X will never happen." People attempt to create rational organizations for their companies and consolidate functions and use common tools; that can orphan a promising piece of software that's the wrong technology. Conway's law: "The structure of software reflects the communications of the humans who created it" is very apt and under-taught.

"Good, fast, cheap" - some cycles, I fear that we don't deliver on any of those three.

We have delivered a client on iOS. Apple's acceptance procedures were, imo, rather rule-bound. There are websites where developers attempt to deduce the real rules (which are a bit different and more complex than the published ones.) It took us a number of iterations to actually get a free app live on the Apple™ store. We've never done an update, although we certainly could. But we did get it out there.

The larger "we" also delivered an app for BlackBerry™ smartphones. The lead developer was pretty unimpressed with that experience; and the app won't run on my new BlackBerry. With BlackBerry, you have to declare the devices you support; of course, you can't declare the ones that you haven't heard of! That project's no longer active, so there won't be a new version anytime soon.

Even with a fairly big team, we continually trade off function for delivery time, leave things for "next releases" that don't always happen, and put major efforts into things that turn out to be just not what the user wants to do.

There are approaches to all of these, but the big hurdles are (guess what) more people issues; people don't always want to change, and people often don't want the ugly realities of needing to choose. We're currently in our second attempt to transform to "agile development"; maybe this one will stick. And maybe not, the notions of "(self) discipline" and "minimize work in progress" are proving to be hard sells.

(In 2010, my team delivered six releases; in 2011, four; and so far in 2012, only one! It would be off-topic to rant too much about why.)

Bruce mentioned "buffer overruns" - among the profound experiences I remember was feeding a string of 9's to a text box on a web page and having the server crash with an exception at program counter 0x39393939. That was found because someone on the Internet had been crashing our server around 9 most mornings - crashing it was even fairly polite, they could have done just about anything to it. New frameworks make this less likely to happen, if used correctly.

#46 ::: P J Evans ::: (view all by) ::: July 09, 2012, 10:49 PM:

broken clients (I mean IE 7)

We just got that one last year. Sometime next year we might get IE8 and Office 2010....

#47 ::: Naomi Parkhurst sees blatant spam ::: (view all by) ::: July 10, 2012, 09:27 AM:

Not even trying. Or else very trying, depending on how you look at it.

#48 ::: Niall McAuley ::: (view all by) ::: July 10, 2012, 09:48 AM:

Corvettes are all very well, but I prefer a frigate, even if rather small by modern standards and often outgunned.

#49 ::: Serge Broom ::: (view all by) ::: July 10, 2012, 09:51 AM:

A Delorean would be nice.

#50 ::: Cadbury Moose ::: (view all by) ::: July 10, 2012, 10:05 AM:

That's rather a Cavalier attitude, isn't it, what if a Corsair suddenly appeared alongside?

#51 ::: albatross ::: (view all by) ::: July 10, 2012, 04:05 PM:


At some fundamental level, I expect that programs will continue to have bugs (and some fraction will be security relevant bugs) for the same reason novels sometimes have plot holes and legal contracts sometimes fail to close all the loopholes some party might try. Languages can make it easier for a small error by the programmer to lead to a security hole (with C's string handling being an extreme example), but nothing can prevent a programmer just not quite thinking things through correctly, and some of the time, the programmer's oversight is security relevant.

#52 ::: Jacque ::: (view all by) ::: July 10, 2012, 04:45 PM:

albatross: Programming errors are by no means a new problem.

#53 ::: Jacque, gnomed ::: (view all by) ::: July 10, 2012, 04:46 PM:

s'il vous plait...?

#54 ::: abi ::: (view all by) ::: July 10, 2012, 05:04 PM:

All the old mainframe jockeys in the conversation are no doubt thinking the same thing I am.


The one-line program that was supposed to do nothing, and still had a bug.

#55 ::: Jacque ::: (view all by) ::: July 10, 2012, 05:19 PM:

abi: Okay, now I'm curious...?

#56 ::: abi ::: (view all by) ::: July 10, 2012, 05:39 PM:

I'm likely to make a mess of this, but here's my best explanation...

This is all in the context of JCL, Job Control Language, which is a way of making sure programs have the correct resources, triggering them to run, and cleaning up after them.

The mainframe operating system (OS/360) is able to do a bunch of useful things as part of running a program, in particular, allocating and deallocating datasets (files). So if you want to allocate or deallocate a dataset without doing anything else (say, to work with later on in the job sequence), the easiest way to do that is to run a program that does nothing, then piggyback the allocation and deallocation on its execution. IEFBR14 is that program.

Unfortunately, when it was done doing nothing, the original version of IEFBR14 did not report that it had done nothing. Because that would be something, which takes more lines than doing nothing. So some tools didn't work well with it.

The final version of IEFBR14 is not one line long, and doesn't do nothing, because what it does is report that it's done.

Corrections welcome.

#57 ::: Jacque ::: (view all by) ::: July 10, 2012, 05:44 PM:

abi: And, as anticipated, your explanation makes more sense (to the uninitiate) than does the Wikipedia article. Thank you!

#58 ::: Lee ::: (view all by) ::: July 10, 2012, 05:56 PM:

abi, #56: So it turns out that what was really wanted there was not "a program that did nothing", but "a program that did nothing but report back that it had executed". Right? Because running a program that literally does nothing is getting into Schroedinger territory.

#59 ::: J Homes ::: (view all by) ::: July 10, 2012, 06:49 PM:

Lee @58.

Almost. What was wanted was a program that reported that it had successfully done nothing.

The original IEFBR14, for reasons that are pretty much incomprehensible to anyone without at least a passing acquaintance with IBM mainframe internals, reported that it had tried to do nothing, and failed.

This failure led to the job being aborted, operators alerted, support staff fetched out of bed, etc.

J Homes.

#60 ::: Jacque ::: (view all by) ::: July 10, 2012, 06:51 PM:

J Homes: The quintessential *headdesk*, wot?

#61 ::: Mycroft W ::: (view all by) ::: July 10, 2012, 08:11 PM:

I am reminded that the linux equivalent to IEFBR14, "true", is 23KB (on my box). (However, note that a lot of that is supporting "true --help" and "true --version", which do *something*, successfully - so violate the spec. That and the fact it's a compiled C program).

As far as I know, there aren't
I believe in the olden days of Unix Unix, true was 1 byte long:


Oh, and while true seems to be unbugged forever, false has had a bug (granted, in outside-the-specification bloat).
What software bloat?

#62 ::: Mycroft W: "Hello gnomes" ::: (view all by) ::: July 10, 2012, 08:13 PM:

I checked the url, really I did. Gratuitous abuse of the &-less-than-sign?

I'm not a great cook. But I will print out the gnomes' convention cards on my new colour laser printer...

#63 ::: P J Evans ::: (view all by) ::: July 10, 2012, 08:53 PM:

That's when you want a Transporter.

(That's the name of the old VW pickup truck.)

#64 ::: Henry Troup ::: (view all by) ::: July 10, 2012, 09:17 PM:

I actually had to re-construct IEFBR14 once - or rather twice, as I faithfully repeated the bug. Circa 1983...

the correct code (from memory) is two instructions

SR 15,15 # zero the result code
BR 14 # return

We'd decided we had to clean up a utility library, and we threw out everything for which we didn't have source. But I thought IEFBR14 was so simple I could just recreate it. Hubris!

The only perfect program, btw, was the original instantiation of Unix™ yes(1) - an empty file. The function was to return success; and that's what the OS did on End of File if no error had been raised. Alas, it didn't last, copyright notices, help functions, and bugs were added.

#65 ::: Henry Troup got gnomed ::: (view all by) ::: July 10, 2012, 09:19 PM:

The gnomes, being creatures of good taste, are rightly appalled by IBM assembler. I offer some fresh-picked gooseberries in atonement.

#66 ::: Mycroft W ::: (view all by) ::: July 10, 2012, 09:25 PM:

Oops, I see what I guess the problem was - undeleted starts of sentences that I decided to rewrite.

It's clear I should now go home. Every spelling flame has a speeling misteak, and every bug report has a bug in it...

#67 ::: Serge Broom ::: (view all by) ::: July 11, 2012, 11:13 AM:

All that jazz..
All that sax and violins...

#68 ::: Henry Troup ::: (view all by) ::: July 11, 2012, 11:13 PM:

Related to this topic is that Windows™ Vista and 7 apparently have a major vulnerability with the "Gadget" interface, so Microsoft is recommending disabling all such desktop toys. PC World story The relevance, of course, is that handhelds and desktops (and laptops) of any stripe are all vulnerable. These systems are quite complex, and it would be a very hard task to determine the origin of all the software on a machine that's been running any time.

#69 ::: Bruce Cohen (Speaker to Managers) ::: (view all by) ::: July 12, 2012, 02:45 AM:

Putting together Henry Troup's remark about the gnomes, and Serge's Facebook comment about the Avengers, I get the IBM dissembler. I assume this was what generated software for the 360-92, the dread CDC-7600 killer1.

1. Feeble nerd joke alert.

Welcome to Making Light's comment section. The moderators are Avram Grumer, Jim Macdonald, Teresa & Patrick Nielsen Hayden, and Abi Sutherland. Abi is the moderator most frequently onsite. She's also the kindest. Teresa is the theoretician. Are you feeling lucky?

If you are a spammer, your fate is in the hands of Jim Macdonald, and your foot shall slide in due time.

Comments containing more than seven URLs will be held for approval. If you want to comment on a thread that's been closed, please post to the most recent "Open Thread" discussion.

You can subscribe (via RSS) to this particular comment thread. (If this option is baffling, here's a quick introduction.)

Post a comment.
(Real e-mail addresses and URLs only, please.)

HTML Tags:
<strong>Strong</strong> = Strong
<em>Emphasized</em> = Emphasized
<a href="">Linked text</a> = Linked text

Spelling reference:
Tolkien. Minuscule. Gandhi. Millennium. Delany. Embarrassment. Publishers Weekly. Occurrence. Asimov. Weird. Connoisseur. Accommodate. Hierarchy. Deity. Etiquette. Pharaoh. Teresa. Its. Macdonald. Nielsen Hayden. It's. Fluorosphere. Barack. More here.

(You must preview before posting.)

Dire legal notice
Making Light copyright 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014 by Patrick & Teresa Nielsen Hayden. All rights reserved.