Hello everyone,
Has anyone ever considered potentially building or launching a platform similar to the Internet Archive but using ActivityPub?
This could serve as a decentralized network to document, preserve, and protect online content from loss, censorship, and other threats, ensuring its availability for future generations.
For those unfamiliar, the Internet Archive is a non-profit that has been preserving digital media and promoting universal access to knowledge since 1996.
It’s famous for services like the Wayback Machine and Archive-It.
Given the importance of preserving digital heritage, especially in the context of censorship and data loss, a Fediverse-based equivalent could fill a crucial role.
The decentralized nature of ActivityPub could provide a robust alternative to centralized solutions.
I’d love to see this kind of project come to life, but, unfortunately, I lack the motivation, time, and energy to take it on alone.
Has anyone else ever considered something similar?
Are there any existing projects that might be interested in this direction?
The issue I see is ensuring that a distributed archive is comprehensive. How do you know what’s missing and needs to be added unless there’s a central coordinating process aware of what everyone already has?
Internet Archive itself has apparently been involved with something called Filecoin, which I assume would solve that kind of issue with ‘blockchain,’ somehow.
https://blog.archive.org/2023/10/20/celebrating-1-petabyte-on-the-filecoin-network/
Is that like the usual blockchains where every computer has to store a complete copy? That would get huge with the Internet Archive.
No, just some metadata:
There are distributed filesystems with redundancy, but the last time I tried something like that, it was extremely slow for both reading and writing. For an offline archive it might be feasible, but you’d have to do a lot of redundancy and error correction to be sure you didn’t lose chunks. Plus, the Internet Archive is so big that even with the data distributed, each participant might have to store a prohibitively large amount.