User:Justauser
Some code I created you may find useful. Or not.
CPoS framework
It is like DPoS, but self-contained, and like qwarc, but doesn't do proper WARCs. Built for one purpose, might work for another.
Note: this version is a bit stale. Try asking me for a newer one if I'm still around. [1]
#Y dashboard
Main, backup location with delayed updates.
An AB-like dashboard for Distributed recursive crawls and other DPoS projects, specially prepared to handle fast and slow projects alike while remaining usable.
Features: full item names, ignores, counters, server-side timestamps, downsampling to a set display rate (2 items per second by default).
To watch another project, use its name as a fragment, e.g. #// dashboard.
Initially forked from tracker dashboard by User:Klea and subsequently developed by me.
sudo git suite
So, you found some Git repos on an endangered server. But for one reason or another, they can't be cloned.
Should you complain about Goddamn Idiotic Truckload of shit and move on? Maybe. But maybe not!
Sudo git suite reimplements "git clone --mirror" out of simpler Git commands, cURL, Wget and hacks to get some fault tolerance.
If everything goes well, applying them results in the same repo as "git clone" would produce. But if something fails, you get a broken repo and a chance to redo some steps manually, where normal clone would burn the evidence and error out.
Make sure you have PHP CLI installed. Some tools also need cURL extension for PHP, "php-curl" in Debian.
"git" tools, remote
Precondition: usable dumb-Git server.
sudo git init
Creates a local bare repo with the same refs as a remote one.
Precondition: an empty directory. [2]
sudo git unpack
Fetches and explodes any packs in a remote repo.
Precondition: a Git repo. [3]
sudo git clone
Gets any missing loose objects over dumb-git.
Precondition: a repo with refs set up to match the remote. [4]
"git" tools, local
sudo git commit
Interactively rebuilds a commit object from the information you might be able to find in a web viewer.
Several common timestamp formats are parsed, obvious typos are detected and missing information is bruteforced. [5]
"cgit" tools
Precondition: a mostly working cgit.
sudo cgit clone
Gets any missing objects from a cgit interface without relying on normal Git protocol.
Precondition: a repo with refs set up to match the remote. [6]
"gitweb" tools
Precondition: a mostly working Gitweb.
sudo gitweb clone
Reconstructs a repo that is only available via Gitweb.
Precondition: an empty directory, a list of refs on the remote in packed-refs format.
Tarball downloads must be enabled. If they are not, use two-URL invocation and provide patch URL instead of the snapshot one - the full repo won't be rebuilt in this case, but enough information to do so will be preserved. [7]
Usage example
So far, tested at:
- https://git.estrogen.zone/[IA•Wcite•.today] - HTTP responses over some length were truncated. Use sudo git init + sudo cgit clone + manual object hunting.
- https://git.main.lv/[IA•Wcite•.today] - HTTP responses over some length were destroyed outright. Use sudo git init + sudo cgit clone + manual object hunting.
- https://g.blicky.net/[IA•Wcite•.today] - HTTP requests ratelimited by Nginx, Git went too fast and failed. Use sudo git init + sudo git unpack + sudo cgit clone with delays in between.
- https://git.zcy.moe/[IA•Wcite•.today] - Git endpoints returned HTTP 500, but cgit worked just fine. Use manual input + sudo cgit clone.
- http://matita.cs.unibo.it/gitweb/[IA•Wcite•.today] - git-daemon broke years ago and HTTP clones were not configured. Use manual input + sudo gitweb clone.
See also
Investigation
Debugging aids: what's wrong with the repo? What's wrong with the server? What file is corrupted?
- git-fsck
- git-rev-list --objects
- git-ls-tree
- git-cat-file
- git-ls-remote
- git(1), search for GIT_TRACE
- gitprotocol-http
Recovery
Replacing the broken parts.
- git-repair?
- Git FAQ entry, outdated
- git-unpack-objects
- git-hash-object
- git-apply
- Software Heritage "raw" API
- World of Code
Usability
If you can't fully recover the repo, the remaining pieces can still be made more useful.
Constructive
Retains original objects - old references are still valid. The repo remains dirty - expect problems.
- git-replace
Destructive
History rewriting - invalidates at least some of the previous references and may destroy data. Leaves a clean but less complete repo.
- Git FAQ entry
- git-repair --force
- git-filter-repo may or may not work
- git-rebase
- git-mktree
- git-commit-tree