[LINK] RFI: Mirroring Complete Web-Sites?

[Roger Clarke]

Link Institute, you're absolutely brilliant!

(For statisticians - there were as many responses off-list as on).

And there were probably about 5 tenable solutions offered. 
Admittedly a couple needed some command-line capability, but I do 
drop in there from time to time, and I can still remember a few of 
the key shell commands without resorting to desperate measures.

As it happened, a colleague (and son - Tony's not the only person 
around these parts with one of those) drew to my attention that he'd 
previously installed SiteCrawler on my machine some years back.  (I'd 
forgotten both it and the circumstances that caused me to need it 
...).

The about-to-be-closed site is now ensconced on my own site, at:
http://www.rogerclarke.com/AEShareNet/

Well, most of it, including all of the site's critical content.

Not the back-end database of course.  But the value of the database 
contents is decaying more rapidly than the licences and the textual 
information about open content licensing in the ed sector.

And the one bit of value-add over the last few years was some FAQs. 
Which some new person put up using ASP;  so they're lost to posterity 
as well.

Thanks very much everyone!

__________________________


At 14:29 +1000 31/8/11, Roger Clarke wrote:
>I need to urgently mirror a web-site, which is about to disappear
>(don't ask, but assume bumbling governmental incompetencies).
>
>I ought to know about things like this, but I don't
>(don't criticise, but assume bumbling Rogerish incompetencies).
>
>I can't get to a database that's in behind the site, but there's a
>great deal of HTML that's well worth rescuing.
>
>I'm a mere user / member of the public, and have no ftp privileges,
>and my first attempts led nowhere (i.e. timed out).
>
>In any case, can an anonymous ftp user do a recursive download?
>
>Is there an easy way to do a bulk download within a browser?
>
>I frequently mirror individual pages in Firefox 3.0.19, using
>File / Save Page As ... / Web Page Complete
>
>But I don't see a way to get it to follow links or directory-structures.


-- 
Roger Clarke                                 http://www.rogerclarke.com/

Xamax Consultancy Pty Ltd      78 Sidaway St, Chapman ACT 2611 AUSTRALIA
                    Tel: +61 2 6288 1472, and 6288 6916
mailto:Roger.Clarke at xamax.com.au                http://www.xamax.com.au/

Visiting Professor in the Cyberspace Law & Policy Centre      Uni of NSW
Visiting Professor in Computer Science    Australian National University