Hey all! Cleaning up my front page, so I am moving information relevant to this downtime to this forum post.
What Happened?
Linode's Newark Datacenter was the target of a fairly large DDOS attack targeting their network infrastructure. This resulted in deteriorated gameplay from 3:30pm-4:00pm and a complete downtime 4:00pm-5:25pm EST.
Was RustOrBust the target?
According to Linode support, our VPS instance was not the direct target of this attack. With that said, it cannot be ruled out that this attack was performed with the intention of knocking us offline.
What has been done to avoid this happening in the future?
Per Linode support's reply, below, they implemented "mitigation tactics so that this sort of attack should not be possible anymore".
Thanks a lot to those who waited around for us to come back online! Our pop has rebounded to it's pre-downtime levels. You guys rule.
-Alex
Most recent reply from Linode support:
Support Ticket 6401299 regarding Linode 'rust (linode2064247)' has been updated by 'jspinosi'
--------------------------------------------------
Hi Alex,
I sincerely apologize for the trouble this has caused for you.
The target of this attack was our infrastructure. Our network engineers have implemented mitigation tactics so that this sort of attack should not be possible anymore and it looks like the Newark datacenter is stable again at this time.
We'll be evauluating the extent of the downtime and providing any necessary credits per our SLA.
If you have any other questions or concerns, please don't hesitate to ask.
Regards,
Jordan
--------------------------------------------------
Please use https://manager.linode.com/support/ticket/6401299 to respond to this ticket.
Thank you,
Linode.com
Timeline of front page posts:
Update 5:40PM EST: Host confirmed DDOS mitigated. Server is back online. Thanks for your patience.
Update 5:30PM EST: I am bringing the server online despite the host not yet confirming all is well. The metrics I am seeing indicate it is over or no longer impacting my VPS instance.
Update 4:48PM EST: We remain completely offline. The server will become reachable briefly, but then drop off again. I will not be trying to bring it back online until my host, linode, posts an all-clear. You can subscribe to notifications relevant tot his downtime here: https://status.linode.com/incidents/91crgqn8d6tl . Still waiting for concrete information regarding whether or not we are the target.
Downtime notice update 4:16pm EST: Per my provider, "We have identified the source of the connectivity issues as an inbound denial-of-service attack and we are working to get this attack mitigated." . At this point there is no evidence we are the target, but it is entirely possible.
Downtime notice 4:00pm EST: https://status.linode.com/incidents/91crgqn8d6tl <-- Really sorry about this folks, but unfortunately my VPS host Linode is experiencing network issues. First time I have experienced this with them; hoping for a quick resolution.
Hey all! Cleaning up my front page, so I am moving information relevant to this downtime to this forum post.
What Happened?
Linode's Newark Datacenter was the target of a fairly large DDOS attack targeting their network infrastructure. This resulted in deteriorated gameplay from 3:30pm-4:00pm and a complete downtime 4:00pm-5:25pm EST.
Was RustOrBust the target?
According to Linode support, our VPS instance was not the direct target of this attack. With that said, it cannot be ruled out that this attack was performed with the intention of knocking us offline.
What has been done to avoid this happening in the future?
Per Linode support's reply, below, they implemented "mitigation tactics so that this sort of attack should not be possible anymore".
Thanks a lot to those who waited around for us to come back online! Our pop has rebounded to it's pre-downtime levels. You guys rule.
-Alex
Most recent reply from Linode support:
Support Ticket 6401299 regarding Linode 'rust (linode2064247)' has been updated by 'jspinosi'
--------------------------------------------------
Hi Alex,
I sincerely apologize for the trouble this has caused for you.
The target of this attack was our infrastructure. Our network engineers have implemented mitigation tactics so that this sort of attack should not be possible anymore and it looks like the Newark datacenter is stable again at this time.
We'll be evauluating the extent of the downtime and providing any necessary credits per our SLA.
If you have any other questions or concerns, please don't hesitate to ask.
Regards,
Jordan
--------------------------------------------------
Please use https://manager.linode.com/support/ticket/6401299 to respond to this ticket.
Thank you,
Linode.com
Timeline of front page posts:
Update 5:40PM EST: Host confirmed DDOS mitigated. Server is back online. Thanks for your patience.
Update 5:30PM EST: I am bringing the server online despite the host not yet confirming all is well. The metrics I am seeing indicate it is over or no longer impacting my VPS instance.
Update 4:48PM EST: We remain completely offline. The server will become reachable briefly, but then drop off again. I will not be trying to bring it back online until my host, linode, posts an all-clear. You can subscribe to notifications relevant tot his downtime here: https://status.linode.com/incidents/91crgqn8d6tl . Still waiting for concrete information regarding whether or not we are the target.
Downtime notice update 4:16pm EST: Per my provider, "We have identified the source of the connectivity issues as an inbound denial-of-service attack and we are working to get this attack mitigated." . At this point there is no evidence we are the target, but it is entirely possible.
Downtime notice 4:00pm EST: https://status.linode.com/incidents/91crgqn8d6tl <-- Really sorry about this folks, but unfortunately my VPS host Linode is experiencing network issues. First time I have experienced this with them; hoping for a quick resolution.
