SourceHut logo
Recently, developers of the popular collaborative development platform «SourceHut" released a detailed report on an incident which resulted in a Service interruption for 7 days due to DDoS attack prolonged, for which the project infrastructure was not prepared.
In their report, they detail that during the initial stage of the attack, the developers They were surprised and did not have enough time to react and counteract the issue on their servers, which led to the upstream provider completely blocking traffic to SourceHut's servers.
For those who do not know about the platform SourceHut, they should know that this It is distinguished by a unique interface that deviates from the look and feel of GitHub and GitLab, but it is characterized by its simplicity, speed and operation without the need for JavaScript. SourceHut offers a wide range of features, including working with public and private Git and Mercurial repositories, a flexible access control system, wiki functionality, bug reporting, built-in continuous integration, chat capabilities, and more. .
About the attack on SourceHut
The developers mention that To ensure the continuity of SourceHut, distributed servers were used in three data centers. The first was used for production configuration, the second for backup, and the third to perform experiments on migrating the infrastructure to a more scalable and fault-resistant service implementation (since the next generation version of SourceHut was in development).
Although Basic services were restored on the third day, (some services remained unavailable from January 10 to 17), it took approximately 9 hours to resolve the access issue to the servers in the main data center after the initial lockdown.
However, the developers faced another difficulty when, the next morning, The attack intensified and began to affect the entire subnet, leading the provider to redirect traffic to the null interface again. Faced with this situation, the developers were forced to urgently start the deployment of the SourceHut infrastructure in another data center using backups. Only after 2 days did they manage to obtain a temporary subnet to access the main servers and continue with service recovery.
To protect against DDoS attacks at the network level, the decision was made to implement an intermediate server in the network of cloud provider OVH, which would provide protection against these types of attacks. All requests would be initially directed to this server and then forwarded to the worker infrastructure.
However, during migration, inevitable difficulties arose, such as errors in the restoration using the rsync utility, network configuration issues and difficulties with traffic redirection, which had to be resolved before DDoS protection at OVH was fully operational. During this time, running servers were exposed to DDoS attack traffic, leading the DDoS protection system to misidentify the servers as the source of the attack.
Despite trying to contact Cloudflare and other DDoS protection services, the cost of protection proved prohibitive. However, the Cloudflare employees later offered free protection as sponsorship for the SourceHut project, although this offer was rejected by the developers, who by that time had already made considerable progress in solving the problem themselves.
Initially, the SourceHut infrastructure migration was planned to be carried out gradually over a period of at least one year, moving the project to servers in another data center. However, due to current circumstances, the migration had to be carried out urgently within 7 days. Fortunately, all SourceHut services were successfully transferred to another data center and the platform has been fully restored.
Finally, if you are interested in being able to know more about it, you can consult the details in the following link.