wikipedia download tool

"Please leave a message at the beep, we will get back to you when your support contract expires."

Moderators: phlip, Moderators General, Prelates

masher
Posts: 821
Joined: Tue Oct 23, 2007 11:07 pm UTC
Location: Melbourne, Australia

wikipedia download tool

Postby masher » Fri Mar 06, 2009 5:42 am UTC

Does anyone know of a wikipedia download tool that would allow me to download an article (and images) and recursively download all articles (and images) that the page links to to a specified depth?

Mzyxptlk
Posts: 513
Joined: Tue Sep 23, 2008 8:41 am UTC

Re: wikipedia download tool

Postby Mzyxptlk » Fri Mar 06, 2009 6:37 am UTC

In the past I have used SpiderZilla, a Firefox plugin, but it only works on versions 2.0.0.x and below.
"Once upon a time, an infinite number of people lived perfect, blissful, eternal lives."

Carnildo
Posts: 2023
Joined: Fri Jul 18, 2008 8:43 am UTC

Re: wikipedia download tool

Postby Carnildo » Fri Mar 06, 2009 7:04 am UTC

masher wrote:Does anyone know of a wikipedia download tool that would allow me to download an article (and images) and recursively download all articles (and images) that the page links to to a specified depth?


wget should work.

masher
Posts: 821
Joined: Tue Oct 23, 2007 11:07 pm UTC
Location: Melbourne, Australia

Re: wikipedia download tool

Postby masher » Wed Mar 11, 2009 4:08 am UTC

Yep, it does/did.

Cheers!

User avatar
'; DROP DATABASE;--
Posts: 3284
Joined: Thu Nov 22, 2007 9:38 am UTC
Location: Midwest Alberta, where it's STILL snowy
Contact:

Re: wikipedia download tool

Postby '; DROP DATABASE;-- » Wed Mar 11, 2009 4:41 am UTC

Is there a "bot-friendly" interface to Wikipedia to retrieve just the text of an article?
poxic wrote:You suck. And simultaneously rock. I think you've invented a new state of being.

User avatar
hotaru
Posts: 1045
Joined: Fri Apr 13, 2007 6:54 pm UTC

Re: wikipedia download tool

Postby hotaru » Wed Mar 11, 2009 4:53 am UTC

'; DROP DATABASE;-- wrote:Is there a "bot-friendly" interface to Wikipedia to retrieve just the text of an article?

there is this...

Code: Select all

factorial product enumFromTo 1
isPrime n 
factorial (1) `mod== 1

Carnildo
Posts: 2023
Joined: Fri Jul 18, 2008 8:43 am UTC

Re: wikipedia download tool

Postby Carnildo » Wed Mar 11, 2009 5:24 am UTC

hotaru wrote:
'; DROP DATABASE;-- wrote:Is there a "bot-friendly" interface to Wikipedia to retrieve just the text of an article?

there is this...

My bots use this.

User avatar
Emu*
Posts: 689
Joined: Mon Apr 28, 2008 9:47 am UTC
Location: Cardiff, UK
Contact:

Re: wikipedia download tool

Postby Emu* » Wed Mar 11, 2009 7:09 pm UTC

Cosmologicon wrote:Emu* implemented a naive east-first strategy and ran it for an hour, producing results that rivaled many sophisticated strategies, visiting 614 cells. For this, Emu* is awarded Best Deterministic Algorithm!

User avatar
'; DROP DATABASE;--
Posts: 3284
Joined: Thu Nov 22, 2007 9:38 am UTC
Location: Midwest Alberta, where it's STILL snowy
Contact:

Re: wikipedia download tool

Postby '; DROP DATABASE;-- » Thu Mar 12, 2009 3:53 am UTC

Carnildo wrote:
hotaru wrote:
'; DROP DATABASE;-- wrote:Is there a "bot-friendly" interface to Wikipedia to retrieve just the text of an article?

there is this...

My bots use this.
Thanks. For some reason I get a 403 trying to use PHP's file_get_contents with the export pages. O_o
poxic wrote:You suck. And simultaneously rock. I think you've invented a new state of being.


Return to “The Help Desk”

Who is online

Users browsing this forum: No registered users and 7 guests