Yes you can download all of Wikipedia and yes it can easily fit in a hard drive. Isn't that amazing? Text is incredibly dense compared to images and video. Around 22 GiB for English Wikipedia alone and 56 GiB for the 10 languages I downloaded.

I also have all of Wiktionary in the same hard drive. It's around 16.4 GiB.

you are viewing a single comment's thread
view the rest of the comments

[–] WarmSoda@lemm.ee 6 points 9 months ago (2 children)

Last time I looked into downloading Wikipedia it said it was 50gb for English text and 100 with images. How'd you get it for half the space?

[–] droning_in_my_ears@lemmy.world 9 points 9 months ago (1 children)

It's only the raw text in json line files. No media and no markup. I think I downloaded a compressed dump then used wikiextractor to extract the text.

[–] AbouBenAdhem@lemmy.world 2 points 9 months ago

Does it include each article’s edit history, talk page, etc?

[–] ace_garp@lemmy.world 2 points 9 months ago* (last edited 9 months ago) (1 children)

The dictionaries for Aard2 are 21gb in .slob compressed format (text only).

[–] WarmSoda@lemm.ee 2 points 9 months ago (2 children)

No idea what that means. But thank you for adding more info.

[–] ace_garp@lemmy.world 3 points 9 months ago

OK yes, some supporting info is: Aard2 is an offline wikipedia app, that uses small compressed data files in .slob format.

[–] intensely_human@lemm.ee 1 points 7 months ago

Slob compression is best visualized as putting a sleeping bag into a stuff sack, except it’s all your possessions and you’re stuffing them into an old Chevy Metro