gogs/templates/outages.tmpl

{{template "base/head" .}}
<div id="outages-div">
	<div class="ui middle very relaxed page grid">
		<div id="outages-links-div">
			<h2> Outages </h2>
			<ul>
				<li><a href="{{AppSubUrl}}/outages#about">About this page</a></li>
				<li><a href="{{AppSubUrl}}/outages#2016-08-31">2016-08-31</a></li>
				<li><a href="{{AppSubUrl}}/outages#2016-01-31">2016-01-31</a></li>
				<li><a href="{{AppSubUrl}}/outages#2015-07-18">2015-07-18</a></li>
				<li><a href="{{AppSubUrl}}/outages#2015-04-30">2015-04-30</a></li>
				<li><a href="{{AppSubUrl}}/outages#2015-04-20">2015-04-20</a></li>
				<li><a href="{{AppSubUrl}}/outages#2015-04-17">2015-04-17</a></li>
				<li><a href="{{AppSubUrl}}/outages#2015-03-12">2015-03-12</a></li>
			</ul>
		</div>
	</div>

	<div class="ui middle very relaxed page grid">
		<div id="outages-list-div">
			<a name="about"></a>
			<h1>About this page</h1>
			<p>Occasionally you'll see <a href="{{AppSubUrl}}/error/error.html">this</a> fellow on your visit to our site. Whenever this happens we've made a mistake. This page is here to help you understand what went wrong and (hopefully) give you a sense that we're constantly improving. If you find any problems using the site please report a bug <a href="{{AppSubUrl}}/hp/gogs/issues">here</a></p>
			<p>We hope you won't have to refer to this page too often. :)</p>
			<br>
			<a name="2016-08-31"></a>
			<h1>2016-08-31</h1>
			<h2> What happened </h2>
			<p> NotABug.org was completey unreachable for most of the 31st of August and the 1st of September. The frontend was not reachable, git pulls and pushes were not possible.</p>
			<br>
			<h2> What caused it </h2>
			<p> Due to a bad DIMM and non-ECC RAM in the server the filesystem of the main gogs repository got corrupted. This had to be restored from backup. No data was lost, however </p>
			<br>
			<h2> How has this been addressed </h2>
			<p>
				<ul>
					<li> A new server with ECC ram was purchased. </li>
					<li> All data was succesfully restored from backups. </li>
				</ul>
			</p>
			<br>
			<a name="2016-01-31"></a>
			<h1>2016-01-31</h1>
			<h2> What happened </h2>
			<p> NotABug.org was completey unreachable for most of the 31st of January and the 1st of February. The frontend was not reachable, git pulls and pushes were not possible.</p>
			<br>
			<h2> What caused it </h2>
			<p> Due to a drive failure in the array the filesystem housing the Git repositories went in read-only mode. During recovery of the situation a admin-error accidentally deleted the root filesystems of both the hypervisor and the NotABug.org VM. Due to this error the SSH key of the server was changed. The new fingerprints are:
			<ul>
				<li> 1024 SHA256:3Gfd/hCr2FAnV34U/LKO93m14yD7JnJOfZPub2sK7No (DSA) </li>
				<li> 256  SHA256:tv7X1j2jOYPgztNC+3KYh2E751I01Y7vHWS83uax8jQ (ECDSA) </li>
				<li> 256  SHA256:BtR8pJSuIPDCUT5DJ0kk/Sxspp9dgWf5p3EE0OUQe3g (ED25519)</li>
				<li> 2048 SHA256:C3P4FLq03YMBgA0ipZrw8AvYLQsq54I1lYUPhPKI5iE (RSA)</li>
				<li> 1024 MD5:c3:27:ea:d8:08:52:f7:c5:5a:68:4d:37:c3:50:32:35 (DSA) </li>
				<li> 256  MD5:44:1a:ae:46:fe:c9:a7:f9:99:77:21:56:bc:02:47:de (ECDSA)</li>
				<li> 256  MD5:fe:ec:8b:51:90:05:bc:3f:dd:21:94:31:46:12:72:d6 (ED25519) </li>
				<li> 2048 MD5:b2:de:dc:04:31:c4:18:85:10:ee:c9:d7:e4:60:57:7d (RSA) </li>
			</ul>Additionally, all the avatars were lost. No repository data was lost, however.</p>
			<br>
			<h2> How has this been addressed </h2>
			<p>
				<ul>
					<li> The systems were reinstalled and restored from backup where possible. </li>
				</ul>
			</p>
			<br>
			<a name="2015-07-18"></a>
			<h1>2015-07-18</h1>
			<h2> What happened </h2>
			<p> NotABug.org web front-end was throwing 500 errors for most of the day. It seems git pushes also did not work.</p>
			<br>
			<h2> What caused it </h2>
			<p> Someone has flooded the Gogs install with thousands of new user registrations. We were effectively suffering a denial of service attack.</p>
			<br>
			<h2> How has this been addressed </h2>
			<p>
				<ul>
					<li> The Gogs VM was restarted. </li>
					<li> The new users were deleted. </li>
				</ul> More work still needs to be done to make this type of attack less likely to succeed in the future. No concrete steps have been taken in this direction, however.
			</p>
			<br>
			<br>
			<a name="2015-04-20"></a>
			<h1>2015-04-20</h1>
			<h2> What happened </h2>
			<p> NotABug.org web front-end was down between 19:00 and 19:30 CEST. It was impossible to view any of the NotABug.org web content. Git pulls and pushes were unaffected.</p>
			<br>
			<h2> What caused it </h2>
			<p> The Gogs software crashed and the monitoring software responsible for restarting it automatically was still faulty. We now know that the problem was that the gogs daemon expects to be able to read the current userid from the $USER environment variable. This only works if the init script is started with the shell runing in bash mode. The init script was called with /bin/sh when running non-interactively (during startup and during the monitor cron job). </p>
			<br>
			<h2> How has this been addressed </h2>
			<p>
				<ul>
					<li> The init script has been adjusted such that it explicitly sets the $USER variable. </li>
				</ul>
			</p>
			<br>
			<a name="2015-04-17"></a>
			<h1>2015-04-17</h1>
			<h2> What happened </h2>
			<p> NotABug.org web front-end was down between 17:00 and 19:00 CEST. It was impossible to view any of the NotABug.org web content. Git pulls and pushes were unaffected.</p>
			<br>
			<h2> What caused it </h2>
			<p> The Gogs software crashed and the monitoring software responsible for restarting it automatically was faulty causing a restart loop.</p>
			<br>
			<h2> How has this been addressed </h2>
			<p>
				<ul>
					<li> The monitoring software has been adjusted such that a restart loop should be less likely.</li>
					<li> Investigations are ongoing to find the root cause of the crash. </li>
				</ul>
			</p>
			<br>
			<a name="2015-03-12"></a>
			<h1>2015-03-12</h1>
			<h2> What happened </h2>
			<p> NotABug.org web front-end was down between 17:00 and 18:00 CEST. It was impossible to view any of the NotABug.org web content. Git pulls and pushes were unaffected.</p>
			<br>
			<h2> What caused it </h2>
			<p> We currently believe that a bad merge of one of Gogs' dependencies caused a problem with the application. At the time the only person who could fix the problem was off at a party. There was no monitoring. Additionally the error page offered no recourse making it seem like we had just gone away.</p>
			<br>
			<h2> How has this been addressed </h2>
			<p>
				<ul>
					<li> The dependencies of Gogs were reverted to known-good versions. </li>
					<li> More people were given access to troubleshoot the Gogs instance for NotABug.org. </li>
					<li> Basic monitoring was put in place which will be improved in the future. </li>
					<li> A <a href="{{AppSubUrl}}/error/error.html">better error page</a> was created with contact information. </li>
				</ul>
			</p>
			<br>
		</div>
	</div>
</div>
{{template "base/footer" .}}