git Gentoo Portage repos on GitHub

git me! The gentoo repos are on github², and I've just discovered them.. mysteriously hidden in the World Wide Web ;)

There is a post on the gentoo forums from 2015, re git/github saying:

Although some developers wrongly still recommend plain rsync-update method, you should never do that, since it is fundamentally insecure (emerge-webrsync is signed by a machine key, but still better). 

You should instead start to sync via git (best do that from github, unless you want to break our infra server).

Essentially, rsync update method is outdated and insecure. The recommendation is to sync update via git, and more specifically github rather than the gentoo infra servers.

GitHub gentoo (without metadata cache, for developers) repo:
http://github.com/gentoo/gentoo

This seems to be a mirror of the gentoo git infra server:
http://anongit.gentoo.org/repo/gentoo

I also found a github mirror³, which what I use now. For anonymous read-only portage sync, this secondary mirror may be more stable and less noisy.

GitHub gentoo-mirror (with metadata cache, for users) repo:
http://github.com/gentoo-mirror/gentoo

Before we change our sync method to git, we need to prepare/clean the portage location. Delete existing files, or move them elsewhere, in case you might have second thoughts ;) If you are not happy with git, you can move them back, and continue with rsync updates.

# mv /usr/portage /tmp

If you don't have enough freespace in /tmp, or don't want to use it, move it elsewhere and remember the location.

My /tmp is on tmpfs, and I prefer that much faster ramdisk for temporary storage. I have enough free space there for now. If you have a similar setup, note that tmpfs will not survive a reboot. If you are worried about that, or don't have enough freespace in /tmp, move it elsewhere and remember the location.

Now we can change our sync method to git.

# vi /etc/portage/repos.conf/gentoo.conf
[DEFAULT]
main-repo = gentoo

[gentoo]
location = /usr/portage
sync-type = git
sync-uri = git://github.com/gentoo-mirror/gentoo.git
auto-sync = yes

Make sure there are no other files in that folder. I had trouble when I renamed my gentoo.conf file to another name in that same folder as backup. I changed auto-sync = no to be safer. Sync wouldn't give me any errors, nor updates!

I previously used the gentoo infra server, before changing over to github, and then to the mirror. Apparently, the gentoo infra servers are not ready to take the weight of the whole world. Apparently, github is safer and more reliable. Apparently, the github mirror³ is more efficient for anonymous read-only sync updates. I was pleasantly surprised that my git sync was so quick after the changeover.

Now that we have prepared, cleaned and emptied our portage for a clean git sync, we can start with it.

# emerge --sync

Once populated, we move the distfiles back where they belonged.

# mv /tmp/portage/distfiles /usr/portage/

Now you can sync again and again.. to your heart's content :) No more worrying about overloaded rsync server warnings, or ending up on the ban list for syncing too much.

It is such a pleasure to sync, and see exactly what has changed where. Such a difference from watching that cruft fly past like a matrix on your screen. Such a difference not to worry about syncing too often and/or receiving warnings. You might miss the filed Change Logs, if you used them. They are no longer needed. Git has much better, automated and less error prone methods. One method is listed below.

$ cd /usr/portage
$ git log sys-apps/portage

Note the efficiency of git sync, and github :) This is what Linus alluded to, when he wanted to have an efficient kernel repository, accessible to the whole world.

If you are happy with git sync, you can delete that portage backup.

# rm -rf /tmp/portage

Another more important aspect is about making open source truly open! Anyone, even without github logins, can view the sources/changes. With a free github account, anyone could make suggestions via a simple pull request⁴. You don't even need to be a developer. You could edit docs.

---

1. nixventure: gentoo portage
2. http://github.com/gentoo/gentoo
3. http://github.com/gentoo-mirror/gentoo
4. http://wiki.gentoo.org/wiki/Gentoo_GitHub