Started By
Message

re: Thanks COMCAST thanks MICROSOFT. My job is dead in the water.

Posted on 11/7/17 at 10:13 am to
Posted by CAD703X
Liberty Island
Member since Jul 2008
86766 posts
Posted on 11/7/17 at 10:13 am to
quote:

Not picking sides or trying to nit pick but you posted


at 2:33pm our phone began ringing off the hook because 100% of our customers had their content session interrupted in the middle of taking a course.

we had reps from Microsoft, our content vendor and the very well-known organization which sponsored the training in an emergency conference for 2 hours.

at roughly 4pm the CDN began to come back. geographically dispersed QA also showed the CDN was starting to show signs of life. by our 4:30PM status update everything appeared to be back online and we sent out a customer communication.

you can make of that what you will, I'm just telling you the facts.

eta from the reddit comments

quote:

I wouldn't put too much faith in the Azure status page. For one thing it was down itself for quite a while today. Also it was reporting everything green until the Azure support twitter got blown up by people reporting outages, then it took about 15 minutes before it started show warnings(!) for only some of the North Central region. Eventually they did put almost all of North Central in warnings, but a complete outage doesn't sound like a warning level event to me.
This post was edited on 11/7/17 at 10:19 am
Posted by t00f
Not where you think I am
Member since Jul 2016
99900 posts
Posted on 11/7/17 at 10:27 am to
Alright I was busy yesterday and did not want to go back and forth on this. We have some customers in Azure (SQL) and their access was slow but not down.

Looking into this a bit more it appears to me Level 3/Centurylink was the main issue and someone at Microsoft did not correctly identify that and starting making changes to fix what they thought was the issue and made things worse.


Posted by mdomingue
Lafayette, LA
Member since Nov 2010
37898 posts
Posted on 11/7/17 at 10:44 am to
Looking back, it did look like I was nit picking.

Ultimately, just wanted to say sorry you had to deal with that crap.

Doesn't matter if it's world wide or just what you have to deal with, it's a royal pain that you have no real control over but have people screaming at you to fix it.

Hope the rest of the week is unevenyful.
Posted by jdd48
Baton Rouge
Member since Jan 2012
22841 posts
Posted on 11/7/17 at 10:55 am to
Hopefully you guys get compensation. Doesn't Azure have a reimbursement backed SLA?
Posted by CAD703X
Liberty Island
Member since Jul 2008
86766 posts
Posted on 11/7/17 at 11:28 am to
quote:

Hopefully you guys get compensation. Doesn't Azure have a reimbursement backed SLA?


we're sorting through that.

its complicated because WE (as the SaaS provider) have an SLA for our customers dicated on the SLA our 3rd party content provider (who is actually the ones using Azure) has with us which they are saying is predicated on the SLA they have with Azure...and of course the games Microsoft is playing and if they have an SLA for Centurylink in place and what was actually down yesterday and for how long is really making this a complicated situation.

Argh! My CEO just wants to know how we're getting compensated for the outage.

So the dust is still settling down on who owes who what. Lots of buck passing.
This post was edited on 11/7/17 at 11:51 am
Posted by SG_Geaux
Beautiful St George, LA
Member since Aug 2004
79566 posts
Posted on 11/7/17 at 11:56 am to
Apparently, yesterdays mess was caused by Level3 misconfiguring a router.
Posted by ell_13
Member since Apr 2013
86704 posts
Posted on 11/7/17 at 1:04 pm to
We are not fans of Level3...
Posted by bountyhunter
North of Houston a bit
Member since Mar 2012
6762 posts
Posted on 11/7/17 at 10:21 pm to
I would look at integrating Azure into your SCOM environment (assuming you are licensed for it). Microsoft actually guarantees their uptime, so if you prove they did in fact have an outage they will pay you back for the downtime. Not saying the 365 environment is perfect, but to be fair most customer outages are caused by their authentication service if they are Hybrid.
Posted by jdd48
Baton Rouge
Member since Jan 2012
22841 posts
Posted on 11/9/17 at 10:54 am to
quote:

We are not fans of Level3...


Just read they're down again. Can anyone confirm?
first pageprev pagePage 2 of 2Next pagelast page
refresh

Back to top
logoFollow TigerDroppings for LSU Football News
Follow us on X, Facebook and Instagram to get the latest updates on LSU Football and Recruiting.

FacebookXInstagram