ANOTHER NATIONAL IT FAIL!!!

[John-in-KC]

It's Monday morning, and the National website is down.

 

Called my Council, asked for the CIO, and his secretary said: It's not just the website, it's all down: ScoutNet, the whole shooting match.

 

National really needs to outsource IT to a competent firm.

 

If it's time to finish your recharter, good luck. You'll need it.

 

[John-in-KC]

It's back...

[emb021]

Yeah, and National wants to take over all the Regional Venturing pages. I guess because they think they can do a better job...

[CalicoPenn]

Well of course they need to outsource because surely equipment failure, or communications failure, or a power failure is clearly National's fault.

 

Now granted, it can be frustrating when we try to do something but discover we can't when we want to do it, but what does it say about ourselves when we immediately jump to the conclusion that it's all National's fault when we don't have any facts other than the service was down at our end.

 

We don't know if the building was affected by a power failure during that time (no power, no servers). When electronics equipment is left on all the time (and servers are left running 24 hours a day), they eventually burn out. It could be that one of the servers failed. Who knows, maybe the HVAC for the server room failed over the weekend causing the server room to overheat which caused the servers to either burn out or shut down so they don't burn out (if they're set up to do that). Maybe a rodent chewed through the T-1 or fiber optic lines over the weekend - servers running fine, but the communications link to the outside world was severed.

 

All this is, or course, speculation, but it seems a bit more realistic that "it was National's fault"

 

Gosh, I don't know - maybe someone can enlighten me on how an outsourced provider wouldn't be affected by beyond control failures, or how we just know that National wasn't affected by any of these possibilities and it was just some guy who pressed the wrong button, or unplugged the server rack, because they just don't know what they're doing.

 

[FScouter]

It's fun to trash the evil "national" eh?

As that other guys says, "take some boys camping".

[sherminator505]

"Those who know what's best for us must rise and save us from ourselves!" - N. Peart

[evmori]

Why was it out? Was it National's IT fault? Did you ask?

[Nike]

The entire Army Air Force Exchange Services net was down world wide this morning too. No credit cards, no POV gasoline access. Their HQ is in....?

 

Yes! Dallas, TX!

[jhankins]

Sounds like a network node failure, not a BSA failure

[Eagle732]

Might have to go back to the good old days when we rechartered with a pencil.

[CA_Scouter]

My company has a 99.99% uptime requirement, this means we get about 53 minutes per YEAR where we can be down. To do this we have:

 

a data center in both SF and NY

full failover replication to multiple servers in each location ( this means that if a one server fails, another one in the same office takes over w/o having to switch to NY )

multiple database nodes ( for the same reason ) in each location

full backup power sources in both locations, in addition to mighty, mighty big UPS devices

replicated data pipes from SF to NY and back, and with multiple vendors in case JimBob from Sprint cuts a cable, we can use BillBob's line from AT&T

 

This is a very simplistic overview btw.

 

We spend millions a year in hardware and software support to achieve this requirement, but we are a news wire release service ( worldwide ) and if we go down, we lose clientbase.

 

Please note that we rarely use NY, its simply there as a backup in case SF goes down. That's a lot of $$ to spend on idle hardware.

 

 

I think National made a decision to not spend that kind of money. This isn't life and limb stuff, eh?

 

 

 

[CalicoPenn]

Recharter with a pencil? But then who can you blame when the point breaks?

[John-in-KC]

CA,

 

Like you, I work in the industry now. My data availability rate is about the same as you, and we're not allowed a redundant data center.

 

Considering my experiences with IT at National, including non-savable fill-in PDFs, ScoutNet recharters that still take 60 days to process, commish visit tracking, I'm convinced an external specialist vendor can do the mission faster, cheaper and better than National ... and that includes online redundancy.

 

 

[NJCubScouter]

I'm sure they are having these problems because they moved to Texas. Companies and organizations in New Jersey never have computer problems.

 

(That was a joke. It's sad that I feel I have to say that, but I know I do.)

 

By the way, when did "fail" become a noun? (As in the title of this thread.) It seems to me that until about, I don't know, maybe a year or two ago, something that "failed" (verb) was a "failure" (noun). Now, quite often, it's just a "fail." (Not when I'm speaking or writing though, I still use "failure.") Maybe this has been developing for years and I just never noticed it, but it definitely seems to be much more common recently. Who decides these things?

[jhankins]

If you want savable fillable PDFs, sent me a note with your email address. I'll make them for you

 

I offered to do this for my council for every form on the website, they declined, citing the inability for the staff to keep up with emails accurately...