[LINK] How Obama Will Use Web Technology

[Stilgherrian]

On 26/01/2009, at 12:13 AM, stephen at melbpc.org.au wrote:
> "Here's a small and nerdy measure of the huge change in the executive
> branch of the US government .. Here's the (Bush) robots.txt file from
> the www.whitehouse.gov website this week:
>
> User-agent: *
> Disallow: /cgi-bin
> Disallow: /search
> Disallow: /query.html
> Disallow: /omb/search
> Disallow: /omb/query.html
> Disallow: /expectmore/search
> Disallow: /expectmore/query.html
> Disallow: /results/search
> Disallow: /results/query.html
> Disallow: /earmarks/search
> Disallow: /earmarks/query.html
> Disallow: /help
> Disallow: /360pics/text
> Disallow: /911/911day/text
> Disallow: /911/heroes/text

That fragment doesn't seem to be particularly limiting to me. Nearly  
all of it is just sensible optimisation for search. You want to  
deliver people to actual content, not search scripts or query forms or  
help pages, so you exclude them from search indexing -- just like this.

The only one which is a possible question-mark is the set of URLs  
ending in "text". I'm guessing they're the text-only versions of  
individual pages, or perhaps the textual content of pages before  
they're wrapped in whatever template with navigation etc before being  
delivered to the user. While there's a possibility this is a bad thing  
because if prevents people seeing low-bandwidth versions of pages (if  
that's indeed what they are), possibly it's actually a good thing  
because you're only giving them pages with their full navigational  
context, not just a bunch of raw txt detached from the rest of the site.

Nothing here indicates a change in the executive branch's mindset,  
just a routine change in the structure of the website. And if this  
came from kottke.org then Kottke ought to know better.

Stil


-- 
Stilgherrian http://stilgherrian.com/
Internet, IT and Media Consulting, Sydney, Australia
mobile +61 407 623 600
fax +61 2 9516 5630
Twitter: stilgherrian
Skype: stilgherrian
ABN 25 231 641 421