Developers recently who are in charge of the package management system Portage (on the Gentoo Linux distribution) announced the release of the stable version of version 3.0.
In which, the main novelty of this new branch presented, is the work that was carried out in the long term on the transition to Python 3 and end of support for Python 2.7 (something that was already seen coming for a long time, since this branch was officially without support for several months)
We have good news! The Gentoo Portage project has recently stabilized version 3.0 of the package manager.
What's new? Well, this third version of Portage removes support for Python 2.7, which has been an ongoing effort in the main Gentoo repository by the Gentoo Python project throughout 2020.
In addition to the discontinuation of support for Python 2.7, another major change that stands out from this new stable branch of Portage 3.0 was the inclusion of various optimizations that they allowed make calculations much faster (between 50% and 60%) associated with determining dependencies.
Interestingly, some developers suggested rewriting the dependency resolution code in C / C ++ or Go to speed up their work, but they managed to solve the existing problem with great effort.
And is that the profile of the existing code showed that most of the time calculation is dedicated to calling the use_reduce and catpkgsplit functions with a repeating set of arguments (the person who led this work mentions that for example, the catpkgsplit function was called 1 to 5 million times).
With the problem detected, mention that to speed up the calculations, caching was applied of the result of these functions by means of dictionaries.
Also, due to a user-supplied patch, updating to the latest version of Portage can greatly speed up dependency calculations by 50-60%. We love seeing our community participate in our software! For more details, check out this Reddit post from the community member who provided the patch. Stay healthy and keep cooking with Gentoo!
Besides that it also notes that the lru_cache built-in function was optimal for this task of caching, but it was only available in Python versions since 3.2.
For backward compatibility, a stub was also added to replace lru_cache, but the decision to end Python 2.7 support in Portage 3.0 greatly simplified the task and made it possible to bypass this layer.
I spent some time profiling Portage with cProfile and vmprof to understand which features were taking the longest. I also generated some flamegraphs from the profiler results, which looked like this. What I noticed was that some functions, like
use_reduceycatpkgsplit, are called very frequently with the same arguments (like, 1 to 5 million times, forcatpkgsplit). I did some experiments to cache the results of these functions in a dictation, and after seeing some good speedups, I submitted a patch to the Portage developer list. Someone suggested using built-in Pythonlru_cachefunction decorator instead, but that's only available in Python 3.2 and higher.
On the other hand, the use of the cache has reduced the operation "emerge -uDvpU –with-bdeps = y @world" on the ThinkPad X220 from 5 minutes 20 seconds to 3 minutes 16 seconds (63%). Tests on other systems have shown a performance gain of at least 48%.
The developer who prepared the change also tried to implement a prototype from the dependency resolution code in C ++ or Rust, but the task turned out to be too difficult, as it required a large amount of code to be carried and at the same time it was doubtful whether the result was worth the effort.
Finally if you want to know more about it About the release note of this stable branch, you can check the details In the following link.