Guys, give me corrections and additions as they come up, and I'll incorporate them into the main entry as quickly as I can.

For the life of me, I can't understand why JC-Hosting refuses to return the database. Would it really be so difficult for them to retain a copy of it? James Cordray said he wanted to keep it as evidence of something or other, but why does he refuse to return at least a copy?

I wonder if he's holding out for a promise that he won't be sued, or some kind of apology.

The forums are Jenna's property, aren't they?

comment thread of my previous post, here and here. More to the point:

Bad link on the second "here"

Sincerely,
BadLinkGuy

As far as I know, none of the AW contingent knows why JC Hosting is holding onto it either. I can't improve on the adjective I've already used: it's a bizarre and ugly situation.

I hope that links to Jenna's plea, and copies of her Paypal button, spread out across the blogosphere the way the links to the Twenty Worst list did.

(And if folks can find it in their hearts and pockets to send some cash to AW that wouldn't hurt either. The sound you hear is popping champagne corks in Boca Raton, Florida, and Frederick, Maryland.)

I've already posted a link on my livejournal.

Thanks, Teresa.

Thank you, Teresa. May your editing pencil never go dull.

My guess is, and this is pure speculation that is being pulled out of the part of me that sits on my chair, that when these folks panicked and took down the site, they just deleted it. Now they have a feeling that maybe that was a bad idea, and they're stonewalling. They had some techs try to recover the data, and they can't, and they're not saying anything because their laywer told them to keep quiet and ride it out, rather than admit to anything.

Just a guess, and I hope I'm wrong.

I'd have to agree with Sean, though I hope I'm wrong too.

Thank you Teresa for all the wonderful support you are offering here. Especially for listing all the efforts in support of Absolute Write.

As for the CafePress store, more stuff to come - I have a whole bunch of sparkly elves working overtime on the graphics for new better stuffs.

I would like to use the 20 Agents list on one (or more) of the products - is there anyone here who can contact Victoria or Ann for me about that permission?

(shouldn't be doing this at work...)

Can Google and the like be contacted directly for help in retriving all the cache info? Is it possible to ask them to set it aside as it were?

Google? No idea.

Ann Crispin and Victoria Strauss have joint weblog. Victoria has a website of her own. So does Ann. You can write directly to Victoria at victoria@victoriastrauss.com, and to Ann at accrispin@accrispin.com.

Go ahead. They're very nice.

It really breaks my heart that this happened. That uncle Jim's thread was pure gold. I'm paypaling now.

Sean? That was my thought when I heard that they'd blown off the first deadline. I've been trying hard not to think that, because it would be such a boneheaded piece of careless vandalism; but when a kid tells you three times they're going to turn in an assignment, and they have an excuse all three times, you have to figure it doesn't exist.

Not sure if they've been looking @ archive.org, but their caches are meant to be permanent. It's not responding for me right now, but it's worth looking there if it ever returns any data:

http://web.archive.org/web/*/http://absolutewrite.com

Google won't help directly; I've already tried asking as a Google consultant.

What AW Water Cooler (the forum) will need in order to deal with all the thousands of cached files is a programmer with parsing skills in a scripting language -- my first thought is Perl, but Ruby or PHP might work -- to This person needs to write several scripts. First a script that:

A. Removes the Google signature text
Chunks the information into separate "records," one for each post, and then each record into the data for the "fields" and puts this into a new delimited file. This would need to match the vBulletin schema, since vBulletin is the forum software
B. Second, a script that goes through the parsed delimited file and inserts the various chunks into the MySQL table, where the data goes.

Assuming I'm not insane, and that this is even viable, and that Jenna decides to go this route.

So if you've got skills like that, please post in this thread, or email me and I'll make sure someone with power authority and wits gets the note.

I should add that this is pro bono, but I suspect would make a nice c.v. entry and I'd be willing to do what I could to make your career flourish.

Eric, Wayback archives fixed pages. The AW message boards were dynamic. There'll be some AW material there, but the majority of it won't have been saved.

I though archive.org was a bit more comprehensive than just static. Not that I can check now though.

As for the scripting bits, it shouldn't be too hard to pull out the post information in a semi structured manner, since all the posts have html ids that correspond to the database post id. (e.g., the post in in a table with an id="post#####", the content in a div with id=post_message_#####, and so on with poster, date posted, and quoting.

The bigggest tricks would be to tidy up the html to produce something that would load into an xml parser and to make sure that all of the forum posts were in the same template. I think htmltidy could probably do the first, and the second might be ok if all the cached pages are from the same template design.

I'd be willing to spend some time on that.

eric

Unmitigated bizarreness. The cynic in me says they're holding back for assurances of some kind. Given the reaction here, I can imagine JC Hosting has been flooded with angry emails and phone calls. Would someone in the business of hosting really just delete a database? I can't fathom it.

I think we need a corollary to the rule: Nothing on the Internet disappears:

Corollary1: Unless someone does something stupid.
Corollary2: But it hides in a billion places.

"A. Removes the Google signature text
Chunks the information into separate "records," one for each post, and then each record into the data for the "fields" and puts this into a new delimited file. This would need to match the vBulletin schema, since vBulletin is the forum software
B. Second, a script that goes through the parsed delimited file and inserts the various chunks into the MySQL table, where the data goes."

I suppose as follows:

Use Google API to get all cached pages, tops out at a certain number of requests but if we get a number of IDs we can go ahead and get all the pages.

I don't have much experience running MySql, I've installed it before and updated tables, converted from legacy encodings to UTF-8 and similar things but it is not my cup of tea. But if someone can provide SQL statements that will work with MySql I can get it to work.

But there may well be others in this community more suitable to this task than me, my main employement is as a specialist in markup technologies for the Danish Government, providing help in the National UBL Ebusiness project, somewhat different stuff.

Contact me through the email rasmussen.bryan at Gmail.com since the bry@itnisk.com one is an old one I still use in the comments so as to provide continuity.

Posted at Forward Motion, and on my LJ.

The only thought worse than they deleted it would be they're hacking and editing it.

Not only Google but Yahoo info as well. Some files that are no longer avaliable at Google can be found on Yahoo and vice versa.

I know enough HTML to do a basic cleaning of the files I retreived before I send them to whoever is going to recreate the forums but I won't dare to go near programming something like what we need here. ;)

In case those incompetenties actually deleted the data, can Jenna sue them for recompensation? It won't replace the lost data of course, but it could help her to pay for new hosting, a professional programmer and such.

Donation made, and yet another post on my LJ detailing the saga. I don't think I've ever posted on AW, but I've lurked there for years and found it very useful. I want it back.

I am in AWE at the job the Absolute Refugees have done in rescuing AW from the jaws of Google oblivion. This is not a technologically oriented community. Oh sure, some of us are techno-geeks in our day jobs, but the majority are not. And yet they have bent to the task with a will that is inspiring to watch. They've stepped up, learned things they thought they'd never learn and done things they could never have imagined doing.

It makes my contribution of setting up the AW Refugee Camp trivial by comparison.

I know of other on-line communities that would have shrugged their shoulders and wandered over to Writers.net to see what was going on there. But not this crew. It says volumes about their commitment to Jenna and the Absolute Write community. People who disparage this as a "lovefest" have no idea how right they are.

I wish I could name everyone involved. I know I'd inevitably miss some. But I just want to send you all a huge THANK YOU!

TW said: The only thought worse than they deleted it would be they're hacking and editing it.

I considered this too – but that would be incredibly stupid. Time stamps, admin logs, etc that’s a lot of data to be “fixing” to hide any tampering. And of course, there are now copies of pages on so many different machines across the planet . . . well, it would just be foolish to try and edit pages to prove they were justified in pulling the site. They can’t be that dumb – can they?

-DJT

bryan said: Use Google API to get all cached pages, tops out at a certain number of requests but if we get a number of IDs we can go ahead and get all the pages.

There are directions here (Windows or Mac OSX) for running a perl script to automatically download pages from the Google and Yahoo caches.

It only allows about 1000 pages to be downloaded at a time, so it would be great if more people were doing it.

They can’t be that dumb – can they?

Hm, wasn't it Einstein who said there were two things that are infinite, the universe and human stupidity, though he wasn't sure about the universe?

DJT, we can only hope they are not that dumb/underhanded. But if they were smart this never would have happend in the first place right? Most likly is that they are trying to wait it out. Ride out the storm till it blows over from some one giving up.

I just had this mental image of IT guys doing what office mangers in desperation of being caught do; shred papers or pack them to home for adjustment. As the husband who works in security points out, people do strange and stupid things in panic that makes them more likly to get caught.

AW loyalists should be pleased to learn that I have, along with several other TERRIFIC volunteers, salvaged over 400 individual threads on the Bewares & Backgrounds Check forum. Those who are regulars there know that not only scammer publishers and agents show up in B&BC--there are many, MANY legit companies who have glowing recommendations and discussions of their current needs. (But yes, I made sure I cached the scam threads first...hate to lose those! :) )

I was reminded today on another site that AW isn't merely the sum of its database, but a community of individuals whose collective knowledge CAN easily rebuild the data from scratch--if necessary. We hope it won't be necessary and the database will be restored to Jenna intact.

I've already made my donation to the rebuild fund, and have linked it on my blog and website. We're grateful for anyone who chooses to donate, since many authors are indeed of the "starving" variety.

Thanks and see you again soon at the new, IMPROVED AW!

Cathy C
AW Romance Moderator

Thanks for keeping the faith, Teresa! Already blogged about this. I have serious PayPal issues, but my check is going in the mail tonight. And continued thanks to Mac, Roger, Cathy, Lori, Charlie, Dawno, Kira, Jim and everyone working feverishly to rebuild.

It's easy to imagine an incompetent admin going to a terminal in a panic and typing "DELETE ALL".

It's less easy to imagine that they also went through every single one of their backups for the last few years and deleted those, as well.

Assuming, of course, that they had backups. Which may be assuming too much.

T.W.: "The only thought worse than they deleted it would be they're hacking and editing it."

Granted that we have yet to see evidence that there's a lower limit to these people's cluelessness, you'd have to be unbelievably stupid to think you could edit posts on a board for aspiring writers and have it go unnoticed.

I know my own writing like a sheep knows its lamb. It wouldn't take much at all -- one or two bits of misplaced punctuation, an alien word choice, a grammatical error I'd never make -- to tell me it had been tampered with. Multiply that by several thousand writers, and you see the impossibility of such a scheme.

On saved bits: I'm hoping people will come forward who've been archiving "Learn Writing with Uncle Jim" for their own use.

MikeB, I've been wondering about that. It seems likely to me that they flew into a stupid panic over Barbara Bauer's threats, and might have deleted the databases -- but what kind of responsible hosting site could have failed to keep backups, and backups of backups?

I take no pleasure in the observation, but these guys may have ruined themselves professionally. Who would want to do business with a site that had no more sense than to let themselves be stampeded by one foulmouthed slag of an agent, dumped a client's data, and turned out to not be keeping proper backkups? I wouldn't touch them with a ten-foot pole.

Lisa, have you asked at the vBulletin modding community?

Does anyone know what they did from the evidence?

I know basic perl. But I don't know nothing about no database stuff or html stripping or whatnot. Sorry. I did forward the request to a perl monger list I'm on with a meager reward for free pizza for everyone at the next meeting. Maybe there's a wizard on that list who can script something up in a couple minutes. Don't know.

Out of curiosity, just how big is this database, anyway? gigs? hundreds of gigs?

"The only thought worse than they deleted it would be they're hacking and editing it."

I would assume that would be a copyright violation unless the TOS says they own the content of anyone they host. If AW stuff showed up under that doppleganger site by the guy's wife, I'd record it and sic a lawyer on them.

If Absolute Write needs a new hosting service, even in the short term, please contact me. I may be able to help.

Wish I could do something about the database...! The whole thing just reinforces the need for Absolute Write to exist.

The database is a couple of gigs.

I'm collecting all the offers to help, and forwarding to People With Authority.

We really do appreciate this folks.

"Backkups" is a word of great organic beauty. Also not a little terror.

The good news is that we've reconstructed the Uncle Jim thread almost in it's entirety--there may be a page or two of the very recent stuff that's missing, but that's it.

We've had a cadre of volunteers finding stuff like Jaws' and HapiSofi's posts, specifically--there are others, but these names just for example--so that they aren't lost.

The Bewares and Background Check stuff we've pulled from the Google-MSM-Yahoo caches, and can reconstruct very nearly in it's entirety.

My deepest and most humble thanks to all the members and volunteers who pitched in and donated hours and disk space to amassing this stuff.

Jenna chose HostGator as AW's new host. Charlie (ChunkyC-the admin) is behind a curtain now hammering the basic structure into place.

I'm not always good at expressing my appreciation and affection for individuals, but I am just overwhelmed by the amount of support for AW. You guys are just... You're what keeps me believing that there is hope for the human race.

Did you guys find someone to parse the cached pages and stuff them into a database? I don't have the knowhow, but I posted a message to some perl wizards trying to beg some help. If you don't need them, I'll let them know.

Preface: this isn't meant to be inflammatory but is a real question from someone that backs everything up a million times if it's important to me...

Why do people post things on the internet that are important to them and not save a copy locally for themselves? Is this a trait that commonly follows writers or just commonly follows everyone and I'm just oblivious?

I mean yeah, the hosting and database all went poof for a really crappy reason (though granted a believeable one, ISPs drop people for stupider reasons). The data, though, should not be in question one way or another imho. They should allow it to be copied over and change whatever their normal rate is for data transfer and that could be that.

The reality is that the site could have poofed altogether without any word at all for any number of reasons. The hosting company could have folded, they could have wiped the databases accidentally or "accidentally", or whatever.

Maybe it's a perspective thing but it just seems really, really weird to me that posting to a site was somehow a "safe" place to keep what's obviously a prized possession.

Zen

Those are reasonable questions, but AW was largely a labor of love, and it grew very very quickly from a pet project with volunteer labor to something ginormous with some high end technical requirements. It was very much a labor of love; most people didn't even realize there was a database, or that it could be backed up, never mind individual posts.

These are naive users, not professisonals, not support people.

Posting is a not trivial thing, for a lot of people. They're not sure, really, where the posts, "are" or "go" and a backup wouldn't occur to them.

Zen, I understand your question--and I don't think you were inflammatory.

Part of it, I suspect, is that people don't think of what they are writing impromptu as all that significant. Then perhaps it turns out rather better than they'd realized, especially in the more creative forums.

I've had a number of letters from grieving poets, for example, whose stuff may well be completely lost--and they'd meant to go and save it for further polishing, but the forums went down so fast that they didn't have the notice or the opportunity to do so.

When there are months or years worth of those accumulated posts, and someone says, "Gee, that stuff is valuable" you might think about backing it up, but it seems pretty safe where it is, really. Frex, I back up a lot of stuff to gmail, because my little nifty keychain gadget is smarter than me.

In terms of the years of accumulated research and correspondence in the Bewares and Background Check forum; that's not really possible to recreate, solo. It's a group effort involving hours and hours of work, stored in one place.

Part of it, too, is that the host is supposed to be doing back ups. That's basic. So the idea that the host would hold our data hostage--it just never occurred to any of us.

Does that sort of answer your question?

Greg, and OG, thanks for the vBulletin modders heads up and passing on the request.

We've had some super offers from ML folk, willing to help. I'm passing them on.

It occurs to me that the volunteers in question work as a group; there are at least two separate tasks, one the parsing and cleaning, and then the filling of the database, but that's not a decision for me to make.

Thanks again.

Hey Gang,
I've been keeping tabs on what's going on...I've been to the camp, but didn't register or post. I'm actually getting work done while AW is down. This whole thing is a big 'ol mess, and some of the PA people are happy (one in particular, who feels this is a 'breath of fresh air')... so we need to get back on the horse. I trust Jenna and Charlie will have us back up to operating speed soon. I posted about this on both blogs a few days ago.

Although, reading those PA forums makes me depressed.

Party on! Back to revisions.

Zen:

I agree with you regarding the personal writing placed in the poetry forum and the share your work forum. But not with everything else.

Most of this stuff only has a value when it exists in its entirety. The Uncle Jim's Writing thread, for example. Who would bother to back up the three sentence question they asked? Or the two-sentence answer? The real value comes from the group of posts... The whole thing which I only posted in a couple times is more valuable than any book on writing I've ever seen.

Maybe I should've saved it. Honestly, the thought never entered my brain. I'm glad to hear others have.

Yeah, Zen has a point there. And it occured to me that most of my blogposts are originals with no backup. *sigh*

I wonder if some techno savvie here has an idea how to save some 200 posts on Blogger in a more elegant way than going to the edit function and copy the code of every single one. BTW I have webspace where I could save it.

The good news is that we've reconstructed the Uncle Jim thread almost in it's entirety--there may be a page or two of the very recent stuff that's missing, but that's it.

I think we're only missing the very last page -- which consisted of posts made the weekend before the plug was pulled. It appears the UJ thread was last cached on 5/19, so I haven't been able to find that one, but I was able to find everything from the end of my Word doc (which ended on March 1) through the second to last page.

And lest the three stooges at PA start celebrating the demise of the Neverending NEPAT -- I've got lots and lots of pages saved and I'm going after more.

Oh crap...backing up my blog posts.

::color drains from face::

Hrm, ok that was a perspective I hadn't thought of. I can see the value in those forums like question and answer threads. No, I would never think of backing those up locally either and they are likely an amazing resource. The aggregate is really where it's at, yes.

Not saving personal stuff, there's probably a "no, no" on both sides...but I agree you don't always know what you have until you've lost it and you don't always realize something is worth saving until later.

Having read JC's explanation (I always try to believe both sides) I think the real villain here is Steph, unfortunatly. She "handled" everything with JC who obviously has hurt feelings, not so much over this particular issue, but over the site as a whole and all of the exceptions and free bandwidth and fixes it was getting. He wrote almost as much about them getting free stuff as the issue at hand.

When Steph suddenly stopped interceding on the site's behalf things went to hell. The site owner obviously didn't know what she didn't know and replied crossly to an already irked-at-AW JC...and...well...boom...

:(

Hey, this thread and the earlier one were just BoingBoinged.

http://www.boingboing.net/2006/05/31/antipublishingscam_s.html

I was farked a month ago, and I got 65,000 hits in one day. I understand BB gets more hits than Fark.

Gabriele - I haven't tried this on a blogging site but there exist various free programs for "harvesting" websites. They are often used for creating mirrors and copies without having direct access to the backend of the site.

Basically it recreates a website (or just one folder or tree of a website) in a folder on your computer. You can then open it in your browser right from the folder as if you were online.

Does anyone have a link to a non-spamware free one? It's been so long since I've had one that I don't know what's out there anymore.

I appologize for being spammy but I just remembered one I used a long time ago: HTTrack

It's available on www.download.com

Please don't take this as an endorsement though, it's been years since I used it and it could be full of spamware or something in it's current version. It also may or may not work for a blog site, no idea, but I bet it would. Just make sure to tell it not to follow external links (or you will store everything you ever linked to!)

When I had used the program it was very functional but wasn't amazingly user friendly, you had to know what you were doing, but I did so...it worked. I don't know if they have made improvements. Can't hurt to try it though I suppose.

If you're on Windows there are utilities for doing a bulk download of blogger and LJ posts for backup purposes. I have yet to see something similar for Mac--though one of the Windows utilities is actually a Perl script so it might not be much of an issue to use Xcode to make it a Mac application.

If you use Blogger and post to your domain, you should regularly archive the various files; being lazy, I just create a zip archive and burn it to cd then email myself at gmail.

MovableType actually has detailed instructions for making a proper backup of the underlying data, so Everything is restored.

I do that, and email myself and burn a cd.

Thank you all so very, very, very much. We're now on day 8 of the "hostage crisis," and I'm fairly certain I'd have lost my mind were it not for the truckloads of support along the way. Thank you for believing in the site's purpose, and for all the offers of help, and for being patient with me.

We are working behind the scenes to get the forums structure back in place, and assemble all the cached pages (there are tons) and figure out how to integrate them.

I've been BoingBoinged before. Let's just say it's visible on the graphed version of the site statistics for the month. Getting linked to by Neil Gaiman is likewise visible from orbit. I've been told by lower-traffic sites that getting linked to by ML shows up as a big spike in their numbers. It's all relative. We're all part of the great trading economy of Interesting Stuff.

The check is in the mail.

Meanwhile, a cure that is worse than many diseases: sufficiently skilled people (which do not include me) can retrieve chunks of text off a hard drive. Law enforcement agencies do it, and so do the we-recover-your-data-for-$$$ companies that advertise in the back of computer magazines. Someone who's desperate for a particular piece might try that.

Good luck, all.

Site harvesters:
httrack, open source? 2nd link is to downloadable exe that doesn't need to be "installed" also available here. You just put it somewhere and double click on it. No registry edits, extra shortcuts, etc.

webreaper, closed source, slightly easier to use, no longer adware.

I have never checked webreaper for spyware/viruses, but I always assumed that it was clean. YMMV

You may need a zip utility like 7-zip to unzip one or more of these.

-r.

Backups. My forum is smaller than AW, a little less than a gig of data, and it's too large for phpMyAdmin to create a dump from. phpMyAdmin was the only access to the database one former host provided. I spent a day creating sql dumps in 15 000-post chunks to get the data out when they gave me notice. There's no way I could have spent that kind of time to do backups on a weekly basis.

Now that I have shell access to my database, making backups is a start-and-forget procedure, and I do backup regularly. But there are a lot of forum admins out there who don't know how to make backups, especially now that "one-click installs" that have become so popular. I've walked several through the basics of making backups in phpMyAdmin.

Gabriele, Blogger has instructions on one of their help pages. A site ripper is also a good suggestion. HTTrack is a freeware (GPL) ripper.

update
Webreaper interfaces with Internet Exploder to use cookies, etc, so if you have crippled, er, limited the security settings on IE, it may not work well. (On my computer, not at all.)

-r.

Hmm. I had a link to the Blogger help page in my last post, but it was considered "questionable content". A google for "blogger export" should get you there.

This just in from Birol -

The time has come to stop retrieving cached threads and
start organizing what we have. At this point, we don't know exactly
what has
been recovered, but due to the Perl script so generously donated by
Shweta,
and the efforts of multiple individuals, we know there will be some
duplication.

In order to keep everyone on the same page and moving
forward, we're asking all Cachers to stop searching for new threads to
recover at Midnight Eastern time tonight.

Stand by for additional information about where and how to
send the threads that have been recovered.

Thanks again to everyone for all your efforts in this.

~Lori

I've been told by lower-traffic sites that getting linked to by ML shows up as a big spike in their numbers.

Yes, it does. The mods at my board thought we were under a DOS attack when you Particled one of our Attack of the Mary Sue stories.

it just boggles the mind that the hosting company has NO regard for the legal implications here... if nothing else, blowing off Jenna's lawyer for the THIRD time is going to not go well if this does go to trial in the future.

I hate to say it, but I agree with some previous posters when they suggested that the database may have just been deleted out of malice/ignorance. And they're afraid to offer their backup copies (if they have them) because it'll show that they did maliciously delete the entire database without warning and certainly without any thought to the legality involved.

as for the individuals involved, well... suffice it to say that they'll never "work in this town again" with their recent actions being documented across the Net.

AW will survive and be even better than before - and will keep tweaking those who seek to scam authors and warn people away from PublishAmerica and their ilk...

:)

Instructions for Blogger "export" here. It's not as simple as some of the other tools make it, but it's doable.

When I write a post for my blog I use Notepad (Editpad, actually, but they're virtually the same) and then c&p into the entry form in Movable Type. That way I have a not-always-perfect but reconstructable copy of the published blog.

For Windows users who want to back up their LiveJournals, I understand ljarchive is a good program. I can't create a link from here, I'm afraid, but you should be able to google it.

For Macintosh users (like me) xjournal has backup capabilities, along with a bunch of other bells and whistles.

I have only been lurking, wasn't a part of Absolute Write (but after reading everything about what's happened I really want to join!), and just wanted to add my sincerest sympathies for what's happened. I really feel for all of you who are hanging and waiting to see whether you've lost a ton of irreplaceable written work.

I frequent a 30-GB board chock full of fictional writing. Two weeks before this, it was nearly lost in a "webhost migration" wherein the new hosts had bought out the old one, and decided they wanted to move 7500 servers from the west coast to the east coast - by physically removing hard drives from one end, shipping them through the mail, then plugging them back in. Without advance notice to anyone, without any kind of migration plan in place to coordinate it, apparently without making backups of anything beforehand, and with no hardware prepared to receive it on the other end. I bet most people can imagine from there what kind of nightmare ensued.

We were in the dark about the whereabouts of our website's hard drive for almost two weeks. We got very lucky when it turned up inexplicably plugged into someone else's server, and that Someone Else was kind enough to return the data. They could've decided to just reformat the disk and not say anything, but they were very kind.

The moral of the stories (both mine and yours) is that one should never trust online servers to do the right thing. There will probably always be incompetent, neglectful, and outright malicious people who can and will do all sorts of mean things to other people's data.

http://www.webhostingtalk.com/ is a good place to look for those sorts of stories (and the one I described occurred as part of a still-continuing saga here).

Thanks everyone for the tips. I don't think Blogger will act the way a certain webhost did, but better safe than sorry, all my travel and writing posts are on the fly, the only stuff I've backed up are a few academic ones.

The Blogger export seems rather complicated - not as in 'I'm too stupid to understand' but as in 'through the backdoor and the kitchen into the main hall'. I'll have a look at httrack - thanks for the warning about webreaper, it would surely mess with my high security settings.

I've got a Europa Domain now and the next days I'll look how to get my blog, forum and website into that one to begin with. The moment I'm going to have a decent income again, I'll also rent a server of my own from 1&1 Germany.

Once again, thank you all so much for your support and the work you are doing on our behalf. My piddling efforts pale next to yours. The new forum is now ready for any data the cache-crew has recovered. We'll soon know just how much we can restore.

It won't be long before we reopen the doors, kids. Hang in there a little longer....

Charlie

PS -- software manuals make poor pillows. ;)

Speculations took this exact same hit in February and survived. It sounds like you're well on the way to recovery, but I'd still be happy to help out any way I can. My contact information in the SFWA directory is current, so please call me if you need me.

Gabriele Campbell says

Yeah, Zen has a point there. And it occured to me that most of my blogposts are originals with no backup. *sigh*

I wonder if some techno savvie here has an idea how to save some 200 posts on Blogger in a more elegant way than going to the edit function and copy the code of every single one. BTW I have webspace where I could save it.

I'm in a similar position with my own blogs. I posted about 500 entries in my AOL Journal before the Fall 2005 exodus from same, only a few of which I had on my hard drive (and those weren't saved in any consistent place or format). I tried to copy them into Blogger, but it's horrendously time-consuming. Seven months later, most of my blogged essays, poems, etc. exist only online. Only the fiction is saved on my hard drive, and not all of that.

I can only imagine what a nightmare it would be for an individual to try to save a local copy of even one forum on an ongoing basis. One tends to assume the data will still be there next time, without such heroic measures.

I have a filthy head cold and am not quite able to follow the updates here, but my partner is a Perl/PHP/database dude who just got laid off, and is willing to volunteer if scripting services are still needed. kmactane at gothpunk dot com will reach him; resume site at gothpunk dot com /~kmactane.

I had him look at Lisa Spangenberg's post at: http://nielsenhayden.com/makinglight/archives/007593.html#128104
and he says this is well within his capabilities. Please email if he can help; I'm too woozy to follow posts here.

For backing up LiveJournals, there's always LJBook (http://lulu.ljbook.com/ ) -- you can make a .pdf file of your LiveJournal and keep that on your hard drive, or go through Lulu.com from there to get a hardcopy version.

Birol and Perks: Roger that. Will stop at 22:00 Mountain.

Everyone's doing so much to bring the forums back. I've taken these last few days to write and catch up, so that when the forums DO come back, I'll have plenty of time to read, lurk, and catch up.

The Uncle Jim thread is one of the most valuable I've ever seen, on any message board, anywhere. Thank goodness that's been saved.

All of us are sending positive thoughts to AW, and Jenna.

I'm off to back up my data now.

D.R.

On a different note, are there others procrastinating by clicking on names of people who wrote interesting posts here to find out if they have interesting blogs? Considering the fact that some people whom I've never seen before have commented on my blog *waves hi and welcome* I suppose I'm not the only one. ;)

Bad writer, no cookies.

Speaking about legal matters, I better get Horatius Ravilla before the treason court in Rome and go on with my NiP. *grin*

Paging Lisa Spangenberg, please note E A Killpack's comment. Thank you. And you too, EAK.

Thanks TNH; You should know I watch ML like a hawk, especially now.

I've been in contact. I'm passing all such offers of help along, up the food chain.

The offers and support are really appreciated. I take a great deal of joy in the fact that the "neutral" net, the Internet as it should be, is fighting back against the other, newer form of the Internet, the one where the DMCA as a poorly conceived but dangerous law, particularly in terms of its use by scam agents and less than comptetent ISPs.

Gabriele:

Here's the way I back up on Blogger (easier than copying all the posts from Edit one by one). Every month of your blog is backed up into an archive page, e.g. "http://lostfort.blogspot.com/2005_05_01_lostfort_archive.html".

Just open the pages month by month in the web browser of your choice and save the complete web page. I've tried it with Firefox in Windows and Linux. In both operating systems, Firefox saves a separate directory with the images on the page if you choose "Save As..." and then "Web Page (complete)" as the format in the next dialog box. Then you can open the page from your hard drive, images and all (useful for my wife, who has a family photo blog). There are also options to save as HTML-only or text, but the pages are probably so small that you might as well just save the whole thing.

You'll have to save once for every month your blog has been in existence (so that might take a while), but after you've done it once, on the first of each month, just save the archive page for the just-completed month before, and you're done.

Not as automatic as a site ripper, but much better than handling every individual post.

To decide where to keep the archives you save, take a leaf from Linus Torvalds: "(Only wimps use tape backup: _real_ men just upload their important stuff on ftp, and let the rest of the world mirror it ;)"

It's good to see so much going on with the caching. I only wish I could have thought to do more. I only got one thread and was only able to give a small donation. I realize it was good to do so, but there's so much more I wish I knew to do.

If this (heaven forbid!) ever happens again, I hope to be somewhat more prepared to help out than I was able to this time.

Still, it is good to know that so much has been done to save such a valuable writer's resource. As a member of AW, and not a mod, I thank you all as well for your help. Were it not for the site in question, I would not have known where to land when I found out the truth about the company that had my book at the time.

I owe a great deal to my fellow writers. The encouragement and help I have received is worth more than gold to me. I hope someday I might be able to do something to, in some small way, pay back those who have been so kind to me.

Dan,
thanks, that should work. I have my blog only since May 2005 so there isn't too much to archive.

Thank you, everyone, who said kind things about the "Learn Writing" thread.

I've put together two pages:

Books for Writers and Book and Movie Recommendations from Learn Writing with Uncle Jim.

These are both Amazon affiliate stores, with the commissions going to Absolute Write. Please feel free to link to them, and even more free to buy stuff from them.

The Book and Movie Recommendations link should be http://www.sff.net/people/doylemacdonald/UncleJim.html, right?

Yes, that's the URL.

Let's see:

Here's from a review of Absolute Write:

The real gem of the AW site is the Water Cooler, or community forum. Here, you can ask questions and get advice from beginning and professional writers of all sorts. There are separate forums for freelancers, screenwriters, children's writers, short fiction writers, novelists, business and copy writers, trade publication writers, poets, song writers, and many more. They are even running a version of American Idol called Absolute Idol -- great fun and a real challenge for those writers who have made it this far!

Novelists will find a real treat in the thread called "Learn Writing with Uncle Jim." Bestselling science fiction/fantasy writer James D. Macdonald started the thread in 2003 to help aspiring novelists learn the trade. The thread has well over 4,000 posts, a separate "undiluted" thread that contains the "meat" of the main thread, and now an index to the topics discussed. Uncle Jim introduces some unconventional theories like "positional chess as a plotting tool" and "all novels are romances." If you're in a hurry, use the index! But the entire thread is well worth a leisurely read over the course of a few weeks or months.

Website Review

Working for a small webhost as I do (basically, it's just me and my boss), and admittedly not knowing the whole story, I do have some sympathies for JC. The amount of bandwidth that the community was apparently using seems quite excessive for a BBS community from my experience, but never having visited Absolute Write I don't know what might have been causing it. If they're a small hosting service, they probably have to pay through the nose for extra bandwidth, and the whole thing was costing them money--plus it was leaving fewer resources left over for everyone else who hosts with them.

I'm also a bit less than sympathetic to AW if it's true that they were even indirectly inciting a mailbombing attack on someone--even a scammer. I take a dim view of denial-of-service attacks of any sort; they end up causing far too much collateral damage.

I do think JC should return the database, but he's probably afflicted with legal-deer-in-the-headlights syndrome, afraid that if he returns it he could be open to liability from Bauer suing him or something, so he feels like the safest thing is to do nothing at all until he has some better idea of what's going on.

Oh, and one more thing...

Typically, webhosts have some sort of "archive manager" function as part of their control panel, which will let you download your own total site backups, independent of any backups that the ISP operator may or may not keep.

If your site contains important, irreplaceable stuff, it's a good idea to use this at least once a week. Once (if) they get their stuff back and get set up on HostGator, it would probably be a good idea for AW to do this at least once a week from now on.

There was no incidement to a mailbombing attack, and as far as I know no such attack took place. Bauer has a long history of making bizarre and baseless claims and threats. When you look up "cartooney" in the dictionary you see her picture.

She has, subsequently, earned herself a googlebombing.

The usual thing that hosts do when they find someone is soaking up bandwidth is to charge them more money.

Robotech_Master:

Hoping this doesn't sound too snarky, but if you are running a small webhosting company, it would be a good idea to figure out now how you will deal with a really successful customer. If you are charging the right rates, and are on the ball, it should be a no-brainer: you charge them based on their usage, you upgrade your connections, you upgrade your servers, and everybody's happy.
,
If you can't afford to upgrade when one of your customers gets really popular, then you don't have a sustainable business model. (Also if JCWebhosting routinely waived the normal charges for AW, as they claim, then they had a problem with their business practices - and they can not reasonably blame their customer for that.)

If you're wondering why I feel qualified to pontificate, I used to run an ISP which offered web hosting among other services. We didn't end up a big national provider, but we never lost money on it either.

The usual thing that hosts do when they find someone is soaking up bandwidth is to charge them more money.

ISPs usually warn the customer in advance that the threshhold is nigh. And you'll note that the ISP's bizarre screed points out that they repeatedly chose not to alert the site that bandwidth was creeping up.

Generally they issue a formal warning, and ask for more money. Money that I feel sure Jenna would have come up with, somehow. She's pretty reasonable.

They can even put a throttle on the port; even I know how to do that, and I'm a Medievalist, not a network engineer.

In any case, ISPs don't hold on to the user's files. Especially when it would take fifteen minutes, tops, to image the 2 gig, verify the image, and burn it to a DVD.

Yeah, I timed it.

It takes me about eighteen minutes to pull a drive and seal the enclosure. I'm not coordinated, so I'd expect better time from someone professional.

Real ISPs don't keep indicating that they will provide access to the data, and then not provide access. The former ISP has done this.

The former ISP has no excuse for keeping the data--data which would allow Absolute Write to actually generate income from the members and content, given the presence of ads and the donations of members. For crying out loud, they could copy the data and still "keep" it.

So now, we're trying to reconstitute the database, but more importantly, to retain the community, and the whole thing will involved lawyers. Lawyers, who, for all their virtues, are expensive, and take a lot longer to do anything than it would take to just copy the damn drive.

They need marriage counselling. Hubby said AW should pay extra for exceeding bandwidth, Stephanie said no, let's be nice to them - at least that's how I read his convoluted post on the other thread.

But probably they're the type who couldn't run a bar if it's situated between a brothel and a porn theatre, as we say here. :)

Robotech_Master commented:

Working for a small webhost as I do (basically, it's just me and my boss), and admittedly not knowing the whole story, I do have some sympathies for JC. The amount of bandwidth that the community was apparently using seems quite excessive for a BBS community from my experience, but never having visited Absolute Write I don't know what might have been causing it. If they're a small hosting service, they probably have to pay through the nose for extra bandwidth, and the whole thing was costing them money--plus it was leaving fewer resources left over for everyone else who hosts with them.

Boingboing is currently linking to this post from JC about AW. I certainly have sympathies for him - I've had the 'fun' of being in a variety of similar situations[0].

James D. Macdonald followed up with:
There was no incidement to a mailbombing attack, and as far as I know no such attack took place. Bauer has a long history of making bizarre and baseless claims and threats. When you look up "cartooney" in the dictionary you see her picture.

I suspect that what he's talking about is actually massive amounts of email and phone calls in -his- direction, complaining about AW being taken offline. Despite not having been planned, if even 1% of the people tracking the AW situation took it into their heads to send email or call, that's a pile of IO to deal with (and good odds that some of it is unreasonable and/or abusive). In fact, from what JC writes, it sounds like there's been a raft of ... antisocial activity ... in the direction of JC Hosting[1].

The usual thing that hosts do when they find someone is soaking up bandwidth is to charge them more money.

It reads as though the two business partners had a failure to agree over what was 'reasonable' in terms of charges - and communication problems with AW on top of that, resulting in more money not being charged[2].

Cynically, some of this is a very familiar story that comes down to a failure to set clear bounds and expectations in a business context resulting in drama, hurt feelings and misery.

From the way that JC's posted, I'd suspect that he feels like he's been badly taken advantage of by a bunch of scammers - rather the way most of the AW audience feel about Bauer, in fact - and Bauer's complaints came as confirmation that he was dealing with a bunch of scammers who were out to screw him over.

It's late, but if there's interest, I can try and write up what I've run into before from the service provider/hosting side of things. It's never black and white *sigh*

[0] Hosting can be a really horrible business, to put it politely - and very often, the hosting provider ends up damn'd however they slice it.
[1] ... and no matter how annoying they might be, trying to hack them, deface personal websites, call repeatedly, email abusively, complain to upstream providers, listing sites as pink or black ... all of these things don't contribute towards a good end solution. Beyond that, when they're strongly associated with one (former) customer, it's hard to be in any respect reasonable[3].
[2] ... although that sounds like there's some business partner issues going on as well.
[3] Yes, as said, I've done time in that industry.

The Cachers were asked to stop searching for data at midnight eastern time Wednesday evening. The questions that naturally followed were: What do you want us to do with it? Where can we send it?

I’m still in the process of clarifying the answers for you. Basically, what we’re going to ask you to do is name the files in a certain format, to make it easy for us to identify them, and then e-mail them to a specified gmail account. I should have those details for you soon.

If you want to continue searching the caches, you may, but as the eighth day of our exile from AW ends, we realize there is a point where we have to redirect our efforts from the recovery of data to the installation of the data that was recovered. We will never be able to find it all. I’ve already tapped a couple of people on the shoulder and asked if they would mind helping with the next phase of the project. In typical AWer fashion they answered, “Yes. What do you need?”

You people are simply amazing.

~Lori

Xeger, I do not believe you will find where anyone with any authority at AW has condoned any hacking attempts or defacing of any online property owned by JC Hosting. Have these events occurred? I am not aware of them if they did. We also did not ask or encourage individuals to call or e-mail JC Hosting or their representatives. What we have said time and time again is that it is a legal matter and to please not to anything which might reflect poorly on Jenna, AW, or interfere with Jenna's legal efforts to recover the data which should have been returned to us as a matter of course.

My partner, who is much techno-geekier than I am, wonders if the following scenario would be of use:

1) Find out JC Hosting gets *their* connectivity from -- their upstream provider.

2) Someone, or several someones, who have lost intellectual property as a result of this debacle notify the upstream provider that *they* have a DMCA complaint against JC Hosting -- that these people have effectively stolen copyrighted material which belongs to them.

That's the thing about tools... they can be used by the good guys too.

Yow. This company is out of Nashville? Nashville has an active SF club, including several writers. Has anyone talked to them to make sure they've heard about this?

From the way that JC's posted, I'd suspect that he feels like he's been badly taken advantage of by a bunch of scammers - rather the way most of the AW audience feel about Bauer, in fact - and Bauer's complaints came as confirmation that he was dealing with a bunch of scammers who were out to screw him over.

Three things to note; both "partners" of the hosting company, where partner should be interpreted as a subsitute for spouse, were active members of the Absolute Write forum.

Secondly: It's not hard to find a great deal of verification about Babara's tactics, which really are more aptly described as "scam" then the actions of an agent (since agents a: don't charge, b: sell books to actual consumer publishers, c: make money for their clients and from their clients).

Thirdly, anyone who caves to a phone call from an overtly emotional caller without a follow-up in the form of a cease-and-desist letter on actual papere, or at least a simulcrum of such . . . well, that's inept if not down right stupid.

Whatever happens next is likely to be expensive. Please go to the link to Jenna's blog, and make a donation. Even if you've never been to AW.

http://jennaglatzer.blogspot.com/2006/05/need-help.html

I find myself wondering. We have all these IT industry qualifications and the technical skills; Microsoft Cewrtified Network Thingummmy and all the rest.

Do any of these cover what might be called professional standards: what your responsibilities are to keep data safe and secure.

I get the impression that we have rather more statute law on that, here in Europe, but the story-so-far leaves me thinking that, whatever JC Hosting did, they don't have that grounding in such responsibilities.

So what certificates should a customer, or employer, be looking for in the USA?

My husband mentioned the Electronic Frontier Foundation, a nonprofit organization that's all about defending people's digital rights. They have lawyers who do pro bono work -- and especially given how Barbara Bauer mentioned the DMCA (which was a fallacy on her part), this legal situation may be right up their alley. Maybe Jenna et al. could give them a shout, see if they can help, gratis?

Robotech_Master:

Typically, webhosts have some sort of "archive manager" function as part of their control panel, which will let you download your own total site backups, independent of any backups that the ISP operator may or may not keep.

I refer you back to my earlier comment about those "archive managers" and their inability to handle large sites. By far the most common interface for databases is phpMyAdmin, which I have never seen set up to cope with anything larger than a few hundred KB. My database when I inherited my site a couple of years ago was around 500KB, and it was already too large for phpMyAdmin to create a dump from.

The parts of AW that were likely maintained by FTP appear to be intact. AW's 2GB database would have taken considerable time and more than a little tech savvy to back up via phpMyAdmin. (I'm assuming the former host followed the usual standard of not allowing shell access to the database. Even with shell, they would not have been able to create a dump in an hour.)

I was never a regular at AW, but I read some threads there, and they were invaluable, especially the Uncle Jim thread.

I don't know which possibility is more infuriating - that the hosting company is so blatantly incompetent as to lose or delete the databases without backup, or that they are so evil (and stupid) as to have the databases and withhold them from the site owners.

I understand from the above that some people are working on the reconstruction of Google-cached data. I'm a perl wizard, and will gladly help with that; if there's any need still - please contact me.

Best of luck to AW and its community.

and no matter how annoying they might be, trying to hack them, deface personal websites, call repeatedly, email abusively, complain to upstream providers, listing sites as pink or black ... all of these things don't contribute towards a good end solution.

I agree wholeheartedly. However, do you have evidence that anyone did anything like this in this case? As near as I can tell, the most that happened was a bunch of people called or e-mailed them once each. This wasn't coordinated. It was just a lot of people who all missed a favorite web site. In at least in the one case, the response she received (for her one piece of e-mail) made her feel like she was about to be sued. (It's in one of the comments on the other Absolute Write thread on this blog.) I understand how he can feel threatened by this, but it's quite a leap to assume that anyone responsible for AW was responsible for some sort of attack on him.

Yes, I understand that all you are doing is taking JamesC at his word, but understand his word has shifted as time has gone on. e.g., he, at first, claimed that Barbara Bauer had nothing to do with the shut down. Now, of course, he claims that she does. For someone who claims that the database is intact, he is behaving rather oddly. Ignoring requests from lawyers does not contribute to a good end solution.

This, oddly, ties into another thread on Making Light. We seem to have the notion these days that to be truly balanced and objective, we must treat all sources of information with equal validity. Thus, we get "he said, she said" journalism. The problem is that all sources of information are not equally valid. JamesC has already demonstrably lied on the record. So unless he points out specific, verifiable examples of AW having violated their TOS or AUP and having been informed of this as per their policy (i.e., cases where Stephanie made the problem go away without involving Jenna don't count) and he can show exactly how he was spammed or which websites were vandalized, I'm not incline to treat what he says with much validity. (In any case, he doesn't actually talk about vandalized web sites. He talks about spammed websites.)

(OTOH, the charges from the other side are that he took down the AW web site and he is holding the data hostage. Both of these have been more than adequately substantiated by now. In terms of credibility, the two sides aren't even close to equal. Actually, if I take him at his word about what happened, the person, IMHO, I think is the source of his problems in this case is not Jenna, but Stephanie, who seems to have, perhaps unintentionally, kept Jenna in the dark about the issues JC Hosting had with AW.)

In this case, where we have Jenna et al. working to reach a good end solution and we have a web hosting company who has done nothing but make unsubstantiated charges, hold data, which by their own service agreement they do not own, hostage, and