Website NetroHost API CDN TheCodingBot Aisuruneko
(19:05 UTC — Jul 25) Resolved – This incident has been resolved.
(09:53 UTC — Jul 25) Monitoring – Finally, coming to a close. We are monitoring the deployment as all websites have now returned and have been reconfigured.
(07:50 UTC — Jul 25) Update – We are actively working on deploying the Netro-related services. Please standby.
(21:10 UTC — Jul 24) Update – NetroHost Users should be able to access their CTs/VMs and perform actions as normal. Note that NetroWeb, NetroCDN, and NetroAPI and other related services by Netro Corporation remain down for the time being. We apologize for the delay.
(18:57 UTC — Jul 24) Update – We recovered what we can and will begin redeploying the server stuff soon. NetroHost Users should be able to access their CTs/VMs soon, if not already.
(17:54 UTC — Jul 24) Update – As we move further into the recovery process, it seems that some profile pictures and banners may be corrupted, if the image was over 1 MB in size. The recovery software used was not able to parse content beyond 1 MB of data for pictures. We will notify you if your user-content (avatar, banner, etc.) was affected, and a “news banner” will most likely appear on the site as well. When more information is available, we shall post them away!
(10:40 UTC — Jul 24) Update – The program used to attempt to recover the data requires a fee (at the least $200), and unfortunately we cannot simply “redeploy”. It may be a bit as we must recover user-generated content (avatars, banners) by hand, using Hexadecimal. This process may take a bit, but we are also going to attempt to get Blue Zenith once a local copy is saved of these hexadecimals.
(08:12 UTC — Jul 24) Update – Okay, so we have some good news - it seems we have successfully been able to see the deleted contents. We will work on exporting and re-deploying Blue Zenith as soon as the full scan is complete or that the structure is fully ready to be saved.
(07:31 UTC — Jul 24) Update – While there isn’t any good news yet, we are working as hard as we can to get this going. To give an update of what has been lost so far are mulitple configuration files related to NetroCorp (including how some parts of the NetroCorp network are proxied), majority of the NetroCDN files (mostly user-uploaded content, like avatars, banners, etc.), legacy NetroHost proxy handler, SSL certificates. We can’t exactly fully recover from a backup, as we would need to rewrite our entire proxy again, but it seems we may need to. We sincerely apologize for the incovience and this isn’t the kind of experience we wish our users to have. If anything, let this be a reminder to make frequent backups! Again, we sincerely apologize and hope to get this resolved as soon as possible. When there are more updates, we shall post them.
(05:12 UTC — Jul 24) Update – We attempted to restore Jahy’s network access and the CTs and VMs for NetroHost users, however we are unsuccessful and would require redoing the entire network to counter this. When more information is available, we shall update.
(01:33 UTC — Jul 24) Update – We are continuing to perform data recovery on some affected services. There is no estimated time at this time as data impacts may have been more extensive than we thought. We will update when more information is available.
(21:16 UTC — Jul 23) Investigating – Netro Corporation US East is experiencing a downtime due to a accidental deletion by one of our network engineers. We are in the process of recovering this affected section and will update when more information is available.