Clunky, outdated technology helped doom Florida’s jobless website — and others — to failure
Tens of thousands of Floridians are weeks into trying to file for unemployment benefits, unaware that a main reason for their endless delays is that the underlying architecture of Florida’s Reemployment Assistance Program website is woefully antiquated for today’s unprecedented economic crisis.
An analysis by the Herald of the state’s Reemployment Assistance Program website, reviewed by half a dozen IT experts, shows a problem hiding in plain sight: dated technology that leaves the newly jobless trying to paddle a canoe against a tsunami.
“Everyone is in the same situation spending most of their time trying to get into this outdated and crappy system,” said Cali Soued, a sales executive in a security technology firm who has tried for almost a month to file for benefits. “Floridians who have become unemployed due to COVID-19 have been left drifting in tumultuous waters with no shore in sight.”
The head of the Florida Department of Economic Opportunity, Ken Lawson, apologized for the catastrophe but neither he nor anyone in his agency has discussed why exactly the website continues crashing, leaving many desperate Floridians trying for two weeks or more to apply for unemployment benefits. Anecdotal evidence from readers suggested some gradual improvement over the weekend.
Relying on IT experts and using tools publicly available on the Internet, the Herald found that the CONNECT system appears to run on IIS Microsoft Servers but, at least until recently, on a version that dates back to when the state renamed and revamped the unemployment benefits program in 2013. And the Microsoft ASP.Net framework used to build the application on those servers appears to be a version that predates the start of the new CONNECT system built during the time of then-Gov. Rick Scott.
“It’s at the end of its life cycle,” said Daniel Castro, vice president of the Information and Innovation Technology Foundation in Washington, D.C., surmised of the CONNECT system.
Neither the Herald nor Castro, who runs the foundation’s Center for Data Innovation, can be sure of the details since the Department of Economic Opportunity has not made them public or discussed them. But online websites such as www.browserspy.com and www.netcraft.com allow anyone to enter a URL address and receive a surprising amount of detailed information, such as the confirmation that CONNECT runs on IIS servers.
Based on this information, Castro and others who have looked at these details conclude that the CONNECT program seems to have been designed without a specific end date in mind. That’s quite different from when the government purchases vehicles, doing so with an expected lifespan for the purchase.
“The same thing is not done for the website, and that is a huge problem,” said Castro, confirming it shows no outward signs of having been upgraded.
Think of it trying to run modern applications on a PC using Windows 7, released in 2009. The result would be slow and unstable performance.
The foundation in August 2018 released a report after reviewing 400 state government websites across the nation. The report ranked states on page-load speed, mobile-device friendliness, security, and accessibility. Florida ranked 44th out of 50 states in mobile friendliness and 41st in accessibility for the disabled. Overall it was “definitely in the lower quadrant,” Castro said.
A software engineer who shared an analysis of CONNECT on the grounds that he not be publicly identified suggested that the CONNECT system dated to a world before larger-scale computing began moving to so-called cloud-based programs.
Rather than buying expensive hardware and software to run, process or store information, many businesses and government entities instead turned to cloud computing providers such as Amazon or Microsoft’s Azure to do this work over the internet.
“The [CONNECT] architecture is poorly designed, with no automatic scalability and reliability,” said the engineer, referring to the ability to add or shrink capacity with cloud-based computing as use of the website grows or falls.
The website also uses a form of load balancing designed by the Seattle-based company F5 Networks Inc., which in theory should let it process web traffic faster but only if there is a more modern architecture.
The outward-facing details of the Microsoft-driven CONNECT system, at least until last week, showed no signs of being connected to Azure. It is widely thought that this may change as the state races to fix its buckling jobless-claims system.
There are other unknown factors that could limit the department’s ability to tap into cloud computing, such as the age of the agency’s mainframe computers, the bulky computer systems used for large-scale data processing, and whether these mainframe computers are used for storing or processing other types of data.
Joseph Reilly worked for a company that helped build the state’s computer networks two decades ago. He was laid off in the tech collapse of 2000-2001 and filed for unemployment. Now an independent contractor, he’s been trying to apply for federal stimulus payments that flow through the state agency and said the system seems to have barely changed in 20 years.
After trying for three days to get into CONNECT, he woke up on the fourth day and remembered that 20 years ago he logged on with a now-antiquated Internet Explorer browser. He tried it, and it quickly worked.
Reilly said that may mean “they haven’t really developed the website beyond the earlier browsers.”
Another IT expert tested access to the CONNECT system for the Herald, also under the condition that he not be identified, and reached many of the same conclusions.
“It does appear to be older technology that is certainly underpowered to handle the kind of demand it is receiving,” said the expert. “Not too surprising as this is probably true for a lot of state systems that haven’t been upgraded in years.”
And a third IT expert with experience in large network computing said Florida’s system was built to process a small number of claims in a state with a traditionally low unemployment rate. It was both a philosophical and practical decision that began under then-Gov. Rick Scott, and his and the subsequent administration of Gov. Ron DeSantis, enjoying strong job growth, built in little headroom for handling larger amounts of traffic in an eventual economic downturn, and certainly not for a pandemic.
Here’s why the flaws in CONNECT matter. Newly released Labor Department data Thursday show that another 170,000 Floridians filed a first-time claim for unemployment benefits in the week that ended April 4.
Remarkably that was a drop of almost 60,000 from the prior week. Nobody thinks fewer Floridians are applying. Thousands of hourly workers in the hospitality, leisure and travel sectors are losing jobs because of coronavirus-related shutdowns and can’t successfully apply for benefits. Nor can the so-called gig-economy workers who were targeted in the Trump administration’s federal stimulus, which is being delivered through state unemployment agencies.
The problems with CONNECT got so bad that Florida last week began manually collecting more than 30,000 paper filings at CareerSource offices and is allowing the unemployed to file claims via FedEx offices, putting both the newly jobless and workers at those locations at risk of infection and thus risking a further spread across the state.
And the Tampa Bay Times reported last Thursday that the DeSantis administration has spent an astonishing sum — nearly $110 million so far — on three contracts with companies just to handle the surging phone calls. These don’t fix the broken website, and cost $32 million more than the initial creation of the CONNECT system, but at least provide phone assistance via at least 1,500 call-center workers to those who are stymied. The phones are key because other problems include the inability to connect via phone to reset a needed PIN number to apply for benefits, and confusion over whether an accepted application actually means someone will receive payments.
Last Wednesday night, the department offered a second, more cellphone- and tablet-friendly website. This effectively eased the process of filling out forms, but those forms still need to then migrate into the broken CONNECT system. The new site is hosted by Amazon Cloud IP, suggesting the agency appears to be belatedly upgrading its ability to handle the surging claims.
Working off error messages shared with the Herald by numerous readers, the second IT expert was able to replicate some of the errors ensnaring Floridians.
“Just poor coding design and improper error handling,” the expert said. “Somewhat surprising it’s something that hadn’t gotten fixed sooner.”
It’s an important observation. Audits conducted by the office of the Florida Auditor General dating back to 2015 have found uncorrected deficiencies in the coding and software development in the CONNECT system. The audits said they could not determine which developer had conducted tests or made changes. These deficiencies were noted in a 2017 report and again in the 2019 audit report, the only one of more than a dozen recommendations that received more than a passing response from the agency.
The Herald reached out to Brenda Shiner, who led last year’s report by the Florida Auditor General, to ask if the problems her team identified explain the buckling of the CONNECT system.
“While our most-recent audit [report No. 2019-183] disclosed that many long-standing findings related to the RA System, also known as CONNECT, had not been corrected, this is not something we are able to specifically answer without conducting an audit of the current problems being encountered,” she said in an email.
Shiner, who is in the IT audit division, confirmed that her auditors have not been asked back in to help assess the current problem but acknowledged that the agency had been warned repeatedly about shortcomings.
“As noted in the summary to our most-recent report on the RA System, many long-standing findings had not been corrected by the Florida Department of Economic Opportunity,” she said.
This story was originally published April 13, 2020 at 4:33 PM with the headline "Clunky, outdated technology helped doom Florida’s jobless website — and others — to failure."
