While fearing for my life the next time I get a prescription...

Data compression algorithms can work, in part, by recognizing repetition in the data being compressed and storing only one copy of the repeated data. Where the input data is grainy, as is often the case with scanned print, "repetition" becomes a matter of judgment; insisting on pixel-perfect matches as the determinant of repetition would result in no matches, and no compression (at least by that part of the algorithm). So, in practice, the machine applies fudge factors to determine whether this grainy part of the image over here is close enough to that grainy part of the image over there to be called a duplicate.

When the fudge factors are set too liberally, similar-looking grainy parts can be mistaken for duplicates. When an erroneous match is made, one or the other of the grainy parts will be assigned as the master part and will be plugged into both places. This can happen to image parts containing text, especially if there is a common visual theme decorating the text. Apparently, the Xerox machines in question are susceptible to this problem primarily when the text in question contains numbers, but not when the text is alphabetical.

There is an update on the situation which includes words like JBIG2, and gets into the specifics of settings on Xerox machines. There are fingers yet to be pointed, and still some question about who knew what when and why certain people didn't understand sooner what was happening. Lawyers may at some point become involved. But at the lowest technical level, it's an image compression issue.

In note that the large copier at work which I use occasionally to scan e.g. a book chapter is set at 200 x 200 as standard and I have to keep resetting it to 300 x 300 (for each document, if I'm copying several, which is a pain, 'cos it resets to the lower res automatically at the end of each document). I'm sure the lower standard setting is to save memory, but it makes for lousy copies, particularly if, as I tend to do, you print two pages to a side* (and double sided of course) to save paper.

*Note: two pages to a side works better with UK/European paper sizes, A4 etc. than with American paper sizes.

I didn't see a claim that standard copying was affected.

Memory/storage space is cheap. Efforts to conserve it almost always come around to bite you in the end.

See: Copier Data Security: A Guide for Businesses

Insignificant. Even if then multiplied by the number of phone calls and emails U.S. citizens make every day, all year long.

We can rely on our data, even in a court of law. Even secret courts of law.

It would be interesting if civilization were to collapse over a software bug as trivial as this.

Architects, contractors, airplane designers, and artillery units had all better be checking what model copier they're using, or things may start falling down, crashing, or blowing up unexpectedly.

I don't believe you'll find many artillery units using Xerox copiers. They tend to favor Canon.

Can't find your gnomed post. Sorry about that.

-- JDM

A useful optional setting to discourage clandestine copying of documents thick with numbers. Could maybe even track the individual copy machine, and when the copies were made.

Our electronic technology is not reliable or secure enough to support this. With a fallback paper record, you have something that neither a glitch nor an attacker can change, which can be used to decide what should be in the electronic records. Without that, you trust the electronic records whether they deserve it or not.

Yes, I agree. This error falls outside of user expectations of a scanner or copier. Expected failure modes include unreadable scans. Readable-but-wrong scans fall outside of that.

I haven't seen anywhere mention if this is the normally configured compression mode of these devices, or whether it's an option the user has to set. Regardless, though, this is not a user error, since it falls outside of reasonable expectations. It'll just influence how many people have been bitten by this error.

Is it possible to determine if your scanned PDFs from one of these devices were scanned with the problematic compression option? If not, nobody who's got documents scanned with these can trust them.

Otherwise, it'll be hard to know if one is among the affected users and thus one will generally have to assume your documents might be wrong.

> A4 to A5 is 70.7%

FTFY.

Going up a size, of course, involves multiplying by √2 (i.e. 141%); going down a size involves dividing by √2 (i.e. multiplying by 70.7%).

Another annoying property of US paper is that there are several different weight scales, all of which are abbreviated to "pounds": bond, index, cover, etc. The rest of the world has just one scale: gsm, or grams per square metre -- a calculation made simple when you know that an A0 sheet is one square metre, or 16 A4 sheets is one square metre.

Further down, he shows scans where '6' has been substituted with '8'.

Wikipedia says that JBIG2 can use either of two methods for encoding data that it thinks is text: pattern matching and substitution, or soft pattern matching. Re the first method, it says "substitution errors could be made during the process if the image resolution is low." The second method stores the differences, so nominally even if it guessed the wrong glyph, the JBIG2 viewer used to view the encoded file would start from the wrong glyph and apply the differences, thus producing something with the correct appearance, which is all you need.

You sometimes see this when copy-pasting text from a JBIG2-encoded PDF: the text looks roughly correct in the PDF, but when you paste into Word or whatever, you see 'I' substituted for '1', or 'O' for '0' or whatever. (This could also be a problem, but not as insidious as the one Mr. Kriesel has identified.)

Scanning paper documents and storing them as PDFs is pretty common practice for EMR, at least in the small clinics I'm familiar with.

I read the original report, and what struck me was that the examples they showed were of small print, only just large enough to resolve some of the distances between a 6 and an 8. It was the sort of thing you could get with a 9-pin dot-matrix printer, a huge relative pixel size. It was marginally readable even without the errors.

If critical records are in print that small, it maybe isn't the Xerox machine that would be your problem.

"Whoever designed the "A" system deserves a prize - and we just take the advantages for granted."

That would be Georg Christoph Lichtenberg for the really cool part of the insight (i.e. seeing the role of root-2), and Walter Porstmann for specifying it to the metric system and making it a standard. (From the Wiki page for "paper size".)

Probably no prizes will be forthcoming, but some people refer to the aspect ratio as the "Lichtenberg ratio." Having your name used as a nickname for root-2 is pretty cool, and better than most prizes. I mean, what prize could I possibly prefer to having some fundamental constant referred to as "the oldster number"?

Pie?

The kind of work she does could be (and sometimes is) the subject of lawsuits and/or "administrative hearings" though it's not the kind that usually depends on the values of a few digits.

Meanwhile on Mefi, a user comments that they've seen this problem in the form of inappropriate keming, and had been wondering what caused that - so it's not just numbers, even if numbers seem to be particularly vulnerable .

Half letter size is actually close enough in aspect ratio to letter that reducing two pages onto one isn't that hard; usually you just have to clip a little bit of whitespace around the edge. The other stuff I mentioned is much more annoying.

That Lichtenberg guy must have been a million laughs. He was the Jon Singer of his day.

For example:

• Home > Honie
• Petroleum > Petroi"Lim
• PROPERTIES > Pl'loPERTIES, or l'flOPER:ms
• Equipment > Equip,;ent, or Eqilipnieht (the latter is obviously the German spelling)

(I find this perversely amusing. If you cock your head and squint just so, you can kinda see how it got there.)

But I do see some numeracy fails:

• 222,195,990 > 222, 195,99\)_
• 6,775,900 > 6,775,90P
• 57,147,790 > 5}; 147:790
• 71,242 > 7i ,242

Interestingly, the errors actually look correct on the PDF, as compared to the paper. It's only when the text is copied out and pasted into another program that the parsing errors show up.

I haven't yet caught it out swapping one number for another. But I haven't actually proofed my processed content against the actual paper yet, either.

But I would never, in a million years, even think about putting out the processed text as "correct," without proofing it first.

The users will be blamed, but shipping a product with a normal setting that can cause this kind of damage is an amazingly bad idea. And I'll bet that there are other copiers and scanners with the same problems.

An OCR'd PDF simply has the text computed by the OCR algorithm placed behind a copy of the original image, so that it is invisible (but still possible to select). When you select some text, you're actually getting the hidden character text, but it happens to line up with the part of the original image.

(Similar scenario that was a topic a while ago: failed redactions of PDFs, where black boxes are just added on top, but the original text is unaltered.)

It's all cool. Xerox knew about this all along. In fact, they warned you about it, in their documentation.

From a user-manual:
“Normal/Small produces small files by using advanced compression techniques. Image quality is acceptable but some quality degradation and *character substitution errors may occur* with some originals.” [asterisks added].

From the IBM spox:
"You are also correct that we have documented in our user guides as well as within our devices that the high compression mode may cause character substitution which means we have known about the potential for this issue. Our design philosophy was to make available a very useful mode that creates small files while at the same time providing information about its limitations."

Both of these quotes are from their blog:

http://realbusinessatxerox.blogs.xerox.com/2013/08/07/update-on-scanning-issue-software-patches-to-come/#.UgfbRmTF2RY

where they also say that they are working on a software patch for this.

I give them some credit for being very honest on the blog. Of course, Corporate Counsel's Office doesn't mind being honest, because they know that they warned about it in the original users manuals. And that warning means that they are shielded from any liability.

Don't you feel more relaxed now? Xerox does!

It would be more honest if the button on the screen said "Sub-Normal" instead of "Normal".

...And now I get to worry about non-nonsensical number swaps. Sigh.

It includes some interesting details not available when this first broke, and among other things I need to retract my statement in comment 10: there are Xerox scanners that, in the factory default setting, mangled text including substituting some characters for others.

Patches now exist for those pieces of hardware, but many places haven't applied them. If you depend on scanned documents that would be made useless by the transposition of one or more characters, ensure that your IT people know of this problem and have applied the appropriate patches.