Special Report: 2020 U.S. census plagued by hacking threats, cost overruns

Special Report: 2020 U.S. census plagued by hacking threats, cost overruns
By Nick Brown

(Reuters) – In 2016, the U.S. Census Bureau faced a pivotal choice in its plan to digitize the nation’s once-a-decade population count: build a system for collecting and processing data in-house, or buy one from an outside contractor.

The bureau chose Pegasystems Inc, reasoning that outsourcing would be cheaper and more effective.

Three years later, the project faces serious reliability and security problems, according to Reuters interviews with six technology professionals currently or formerly involved in the census digitization effort. And its projected cost has doubled to $167 million — about $40 million more than the bureau’s 2016 cost projection for building the site in-house.

The Pega-built website was hacked from IP addresses in Russia during 2018 testing of census systems, according to two security sources with direct knowledge of the incident. One of the sources said an intruder bypassed a “firewall” and accessed parts of the system that should have been restricted to census developers.

“He got into the network,” one of the sources said. “He got into where the public is not supposed to go.”

In a separate incident during the same test, an IP address affiliated with the census site experienced a domain name service attack, causing a sharp increase in traffic, according to one of the two sources and a third source with direct knowledge of the incident.

Neither incident resulted in system damage or stolen data, the sources said. But both raised alarms among census security staff about the ability of the bureau and its outside security contractor, T-Rex Solutions, to defend the system against more sophisticated cyberattacks, according to five sources who worked on census security, as well as internal messages from security officials that were reviewed by Reuters.

Among the messages, posted on an internal security registry seen by Reuters, was a note observing that T-Rex’s staff lacked adequate forensic capability as recently as June of this year. “In the event of a real-world event such as a significant malware infection,” the team would be “severely limited in its capability to definitively tell the story of what occurred,” the message said.

One of the sources with direct knowledge of the hack involving Russian IP addresses described the internal Census Bureau reaction as a “panic.” The incidents prompted multiple meetings to address security concerns, said the two sources and a third census security source.

Census Bureau spokesman Michael Cook declined to comment on the incidents described to Reuters by census security sources. He said no data was stolen during the 2018 system test and that the bureau’s systems worked as designed.

The work of Pega and T-Rex is part of the bureau’s $5 billion push to modernize the census and move it online for the first time. The project involves scores of technology contractors building dozens of systems for collecting, processing and storing data and training census workers for the once-a-decade count. T-Rex’s security work is projected to cost taxpayers up to $1.4 billion, according to the census budget, making it the largest recipient of the more than $3.1 billion that the bureau set aside for contracts.

The problems with Pega and T-Rex reflect the Census Bureau’s broader struggle to execute the digitization project. The effort has been marred by security mishaps, missed deadlines and cost overruns, according to Reuters interviews over the past several months with more than 30 people involved in the effort.

“The IT is really in jeopardy,” said Kane Baccigalupi, a private security consultant who previously worked on the census project for two years as a member of the federal digital services agency 18F, part of the General Services Administration. “They’ve gone with a really expensive solution that isn’t going to work.”

The potential costs of a hacking incident or a system failure go beyond busted budgets or stolen data. A technological breakdown could compromise the accuracy of the census, which has been a linchpin of American democracy since the founding of the republic more than two centuries ago.

The U.S. Constitution requires a decennial census to determine each state’s representation in Congress and to guide the allocation of as much as $1.5 trillion a year in federal funds. Census data is also crucial to a broad array of research conducted by government agencies, academics and businesses, which rely on accurate demographic statistics to craft marketing plans and choose locations for factories or stores.

In a worst-case scenario, according to security experts, poorly secured data could be accessed by hackers looking to manipulate demographic figures for political purposes. For example, they could add or subtract Congressional seats allocated to states by altering their official population statistics.

The Census Bureau says its information-technology overhaul is on-track. Systems supporting initial census operations – such as creating its address database and hiring workers – are “fully integrated with one another, performance-tested, and deployed on schedule and within budget,” bureau spokesman Cook said.

Cook said that the bureau had conducted a “bug bounty,” a bulletproofing practice in which benevolent hackers are invited to search for vulnerabilities. He called the effort successful but declined to provide details for security reasons.

Lisa Pintchman, a spokeswoman for Cambridge, Massachusetts-based Pega, said the company was selected through a “very rigorous process” and stands by its work. T-Rex, headquartered in Maryland, declined to comment.

The escalating costs and reliability concerns for Pega’s front-end website have prompted the bureau to consider reverting to an in-house system, which remains under construction as a backup, according to three technology professionals involved in the census project. Census spokesman Cook confirmed that the in-house system, called Primus, would be available for use if needed next year.

This exclusive account of the Census Bureau’s technology troubles comes after government oversight agencies have chronicled other security problems, delays and cost overruns.

The Government Accountability Office (GAO), the fiscal watchdog for Congress, has said the 2020 census is at high risk for a breach or system outage that could prevent people from filling out surveys. The GAO has also said the bureau’s information technology systems won’t be fully tested before the census kicks off for almost all Americans on April 1, 2020, and that 15 of the bureau’s systems – including Pega’s data collection mechanism – were at risk of missing development deadlines ahead of the census.

The Inspector General of the Department of Commerce, meanwhile, in October announced plans to audit the bureau’s technology operations, months after identifying mismanagement of its cloud data-storage system that left it vulnerable to hackers.

Cook declined to comment on the audit but said the bureau is poised to “conduct the most automated, modern, and dynamic decennial census in history.”

The effort to move the census online aims to streamline the counting process, improve accuracy, and rein in cost increases as the population rises and survey response rates decline. Adjusting for 2020 dollars, the 1970 census cost $1.1 billion, a figure that rose steadily to $12.3 billion by 2010, the most recent count. The 2020 tally is projected at $15.6 billion, including a $1.5 billion allowance for cost overruns.

The bureau’s technology woes mounted outside the limelight, as Washington focused on the Trump administration’s push to add a question asking census respondents if they were U.S. citizens, part of a larger effort to curb illegal immigration.

The president abandoned that effort in July after the U.S. Supreme Court rejected it, cheering civil rights groups who had worried it would dissuade immigrants from responding and cost their communities political representation and federal dollars. Still, an October 18 study by the nonpartisan Pew Research Center found that more than one-fifth of Hispanics say they may not participate in next year’s census, compared to 12% of whites.

Graphic: Why the Census Matters, https://graphics.reuters.com/USA-CENSUS/010092JZ39N/index.html

‘SINGLE POINT OF FAILURE’

The census technology overhaul got off to a late start, in part because Congress gave the bureau less funding than it requested for most of the decade. Pressed for time, bureau leadership at times prioritized speed over security, according to four people familiar with the bureau’s security operations.

New technology systems, they said, were tested in settings that were vulnerable to hackers despite carrying unresolved risks that had been identified by the bureau’s in-house security team. The testing was authorized by bureau leadership and supported by T-Rex, over the objections of the in-house security officials, who wanted the vulnerabilities fixed first, three of the people said. It stoked internal tensions that ultimately led one security boss to quit his post, the people said.

The Census Bureau’s Cook declined to comment on whether the testing was done over the objections of in-house security officials but said that the bureau follows a strict protocol to minimize risk.

The bureau began rolling out its technology plans in 2014, promising a technological tour-de-force with 52 separate systems. Twenty-seven of them will be used for collecting census data, which include building the website where respondents submit forms and the tools used by door-knockers tasked with nudging stragglers.

Most of the Census Bureau’s $5 billion in technology spending has gone to seven main contractors, who together have tapped another 41 companies as subcontractors, according to public presentations by the Census Bureau in 2018.

Within months of the rollout, government advisors from two outside agencies – the U.S. Digital Service and 18F – began warning officials off the sprawling approach, according to Baccigalupi and five other people familiar with the discussions. The outside advisers urged a simpler system, one that would be easier to defend against hacks and glitches.

The Digital Service was created in 2014 by President Barack Obama after the troubled launch of Healthcare.gov, the website meant to allow Americans to sign up for health insurance under Obamacare. Design flaws left the site overwhelmed by higher-than-expected traffic and prevented many users from registering for weeks. Digital Service officials saw the 2020 census as a potential repeat of that fiasco, two of the people said.

The General Service Administration’s 18F unit – named for the address of its Washington, D.C. office – functions like a private-sector consultant and is paid by agencies seeking technology help.

18F declined to comment for this story, and the Digital Service did not respond to requests for comment.

The debate between Census Bureau leadership and its advisors from the Digital Service and 18F focused on two broad approaches to software production: monolithic versus modular.

A monolithic framework – like the one envisioned by Census Bureau officials – bundles different functions into one system. In the case of the census, that could mean a system that allows people to answer the survey on a website, translates incoming responses into data and stores it. Monolithic systems can be easier to build, but critics say they become hopelessly complex when something goes wrong. A problem with one function can shutdown the whole process.

“It’s a single point of failure,” Baccigalupi said.

In a modular system, by contrast, engineers build different pieces of software for each function, then write code to allow them to interact. While it’s more challenging to move data through different components, the risk of a system collapse is much smaller. If one function breaks, others can still work while it’s repaired.

Census officials brought in 18F and Digital Service consultants on long-term secondments to help with aspects of the project but largely ignored their recommendations to take a more modular approach, said 18F’s Baccigalupi and Marianne Bellotti, a former agent at the Digital Service who consulted on the project in 2017.

“I told them pretty consistently in 2017: If you suffer a denial-of-service attack, I’m not sure your architecture can withstand it,” Bellotti said.

In a denial-of-service attack, a hacker tries to prevent legitimate users from accessing a program, often by overwhelming it with more connection requests than it can process. Any extended outages during the census would reduce response rates, compromising the accuracy of the data and making it more expensive to collect.

Cook, the Census spokesman, did not comment on why the bureau chose a more monolithic approach but said the consultants recommending against that path did not fully understand its systems.

“18F and USDS looked at portions of our systems and provided recommendations, but neither group had an overall understanding of how those systems integrated or their capabilities,” Cook said.

Graphic: Census costs soar to more than $15 billion, https://fingfx.thomsonreuters.com/gfx/editorcharts/USA-CENSUS-TECHNOLOGY/0H001QXF693K/index.html

RISING COSTS

Bellotti and Baccigalupi say they told the bureau repeatedly in 2016 and 2017 that Pega’s technology wasn’t well-suited to its central tasks – building the self-response website and the mobile applications to be used by census door-knockers. Pega’s code, they argued, would require so much customization that the final product would be slow and prone to glitches.

“If you want to build the fastest car in the world, you build that car from scratch,” Baccigalupi said. “You don’t try to customize a tour bus until it’s the fastest car in the world.”

The Census Bureau’s outside advisers from Carnegie Mellon University’s Software Engineering Institute shared the concern and told the bureau in a 2016 memo, which was reviewed by Reuters, that commercial products such as Pega’s “are not designed to meet an organization’s specifications.”

Neither the bureau nor Pega commented on the assertion that the need for customization made the system expensive and unreliable.

Before hiring Pega, the bureau already had a workable system for data collection, built by in-house staff, Baccigalupi said. Starting in 2014, small teams had fashioned prototypes for online responses and mobile apps that seemed to work. The online response prototype, known as Primus, had been built at little cost beyond the salaries of the half-dozen or so coders.

The in-house systems were tested, and Primus was used in a real-world setting during smaller surveys conducted by the bureau. All performed well, John Thompson, who served as Census Bureau director from 2013 to 2017, said in an interview.

In a 2016 public report explaining its choice to go with an outside contractor, the bureau called Pega’s product a “commercial off-the-shelf solution” that could work with minimal alterations. Pega would do what Primus and the in-house mobile apps could do, but cheaper, with an estimated price tag of $84.5 million, compared to the $127 million forecast for building in-house. Pega would also supply other key functions, such as transferring user responses to data storage.

The reality was messier. Pega’s off-the-shelf solution has required so much modification that it has become “unrecognizable,” said one former Census Bureau official involved in the contracting process. In January 2018, the bureau nearly doubled Pega’s cost estimate to $167.3 million. It has spent about $149 million so far.

Contract documents reviewed by Reuters showed about $121 million of Pega’s contract has gone toward “contracting services,” a category that two former bureau contracting officials said typically refers to the labor required to write and customize code. The figure is more than 13 times Pega’s initial estimate for contracting services.

The bureau did not comment on the escalating costs. Pintchman, the Pega spokeswoman, said the work is “on budget” and that “any changes in estimates would be a result of changes in project scope as well as the Census Bureau identifying additional opportunities for us to add value.”

Thompson, who ran the bureau at the time it decided on Pega, described the decision as a “tough call.” While Thompson and his team viewed Primus as capable of scaling up for the 2020 Census, he said the prospects for scaling up the in-house prototypes for census-worker mobile apps were less certain.

As Pega’s problems have become more clear, Census officials have considered reverting to Primus, the in-house system, for data collection, said three sources familiar with the bureau’s thinking. As recently as this summer, they were instructing employees “to build Primus out, in case it was needed,” said one of those people.

SECURITY INCIDENTS

The only full-scale test of the system took place in Providence, Rhode Island, last year. The bureau conducted a kind of dress rehearsal – essentially a mini-census, with respondent data collected and stored online.

That’s when the system was accessed from IP addresses in Russia, the two census security sources said. Other hackers launched a domain name system attack on the website, which one source described as similar to a denial-of-service attack.

The domain name system attack was not as worrisome as what it revealed about the abilities of T-Rex to respond to such a threat, according to five people involved in census security.

T-Rex staffers “didn’t know how to access the cybersecurity defense tools that were in place, and they didn’t know what to look for,” said a person familiar with the operation. This source added that the bureau had purchased a license to use forensic-analysis software, called EnCase, to investigate hacks more than a year earlier, but T-Rex had yet to fully integrate EnCase into the security system when the security incidents occurred.

T-Rex’s security work had encountered trouble early on. The GAO reported that, by June of 2018, Census’ Office of Information Security (OIS) had flagged more than 3,000 security compliance deficiencies, 2,700 of which were related to components being developed by T-Rex.

OIS voiced concern over the flags and recommended addressing the bulk of them before testing, according to two security officials familiar with the matter. But bureau leadership authorized live-testing of the systems anyway to keep the project on schedule, the people said. The bureau’s Office of Information Security chief, Jeff Jackson, quit his post in October out of frustration over his office’s lack of influence on the project, two sources familiar with the matter said. Jackson did not respond to requests for comment.

A June report by the Department of Commerce’s Office of Inspector General called attention to other snafus. It revealed that, for a prolonged stretch in 2018, the bureau lost the codes needed to gain unrestricted access to its Amazon-based cloud data-storage system. Without the codes, the IG reported, the bureau could not have stopped a hacker from accessing or destroying data stored in the cloud.

The IG, in an October 17 letter to Census Director Steven Dillingham, said it would “immediately” begin auditing the bureau’s technology to “determine the effectiveness of security measures.”

Baccigalupi, the former 18F consultant, called the project’s problems to date “infuriating” given the high cost to taxpayers, and said the bureau’s internal staff could have built the systems better and cheaper.

“Those teams are eager to do it,” Baccigalupi said, “and demoralized to see bad and expensive software going out instead.”

(Reporting by Nick Brown; Editing by Richard Valdmanis and Brian Thevenot)

Exclusive: U.S. Army forms plan to test 40,000 homes for lead following Reuters report

FILE PHOTO: Professor Alexander Van Geen, Research Professor of Geochemistry at Columbia University, tests lead samples from Fort Benning, Georgia at the Lamont-Doherty Earth Observatory in Palisades, New York, U.S. March 29, 2018. REUTERS/Mike Wood/File Photo

By Joshua Schneyer and Andrea Januta

NEW YORK (Reuters) – The U.S. Army has drafted a plan to test for toxic lead hazards in 40,000 homes on its bases, military documents show, in a sweeping response to a Reuters report that found children at risk of lead poisoning in military housing.

The inspection program, if implemented, would begin quickly and prioritize thousands of Army post homes occupied by small children, who are most vulnerable to lead exposure. Ingesting the heavy metal can stunt brain development and cause lifelong health impacts.

The lead inspections would cost up to $386 million and target pre-1978 homes to identify deteriorating lead-based paint and leaded dust, water or soil, according to the military documents.

A draft Army Execution Order says the program’s mission is to mitigate all identified lead hazards in Army post homes in the United States. In homes where dangers are detected, the Army would offer soldiers’ families “temporary or permanent relocation” to housing safe from lead hazards, it says.

The Army’s mobilization comes after Reuters published an investigation on August 16 describing lead paint poisoning hazards in privatized military base homes. It documented at least 1,050 small children who tested high for lead at base clinics in recent years. Their results often weren’t being reported to state health authorities as required, Reuters found.

Behind the numbers were injured families, including that of a decorated Army colonel, J. Cale Brown, whose son JC was poisoned by lead while living at Fort Benning, in Georgia.

The article drew a quick response from lawmakers, with eight U.S. senators demanding action to protect military families living in base housing.

The Army’s planned response is laid out in military documents, including the draft Execution Order, minutes from a private meeting attended by top Army brass, and other materials.

One priority, detailed by Under Secretary of the Army Ryan McCarthy in an August 22 meeting, is for the military’s response to counter any sense “that we … are not taking care of children of Soldiers and are not taking appropriate action quickly enough,” meeting minutes say. “The Army will remain focused on the actions to assess, inspect, and mitigate risks to Soldiers and Families,” the minutes say, citing McCarthy and Vice Chief of Staff General James C. McConville.

Army spokeswoman Colonel Kathleen Turner acknowledged plans are being formulated but said no decisions have been made. “Out of an abundance of caution, we are going above and beyond current requirements to ensure the safety of our soldiers and their families who work and live on all of our installations,” Turner said in a statement. “We are currently evaluating all options to address these concerns.”

Old lead-based paint becomes a poisoning hazard when it deteriorates, and poor maintenance of military base homes can leave legions at risk. About 30 percent of service families – including some 100,000 small children – live in U.S. military housing owned and operated by private companies in business with the military.

There are nearly 100,000 homes on U.S. Army bases, and the lead inspections are expected to focus on the approximately 40,000 built before a 1978 U.S. ban on the sale of lead paint.

The plans depart from guidance that appeared on the Army Public Health Center’s website as recently as last week, which “discouraged” lead-based paint inspections in Army homes. The website has since been updated and omits that language.

Under the plans, the documents show, the Army would:

– Inspect all pre-1978 Army family housing units nationwide, including visual lead-based paint assessments by certified personnel, swipe-testing for toxic lead paint dust, and testing of tap water. Some homes will also receive soil testing. This phase alone, described as “near-term actions,” will cost between $328 million to $386 million, the Army’s Installation Services director estimated.

– Temporarily or permanently relocate families when hazards are found. “If a Family or Soldier are concerned with potential negative impacts from lead; the U.S. Army will offer them a chance to relocate to a new residence,” the documents say. “We must do everything we can to maintain that trust.”

– Conduct town hall meetings on Army posts to address residents’ lead concerns. The Army intends to do so with “empathy,” the meeting minutes say. “Tone is key and can be just as important as the actions we take.”

The documents leave some questions unanswered. They don’t say how long it would take to inspect all 40,000 homes. Also unclear is whether the Army has funds immediately available for the program, or would need Congressional authorization to set them aside.

The Army would ensure that the private contractors who operate base housing “are meeting their obligations” to maintain base homes, the documents say and would require them to show compliance with lead safety standards through independent audits.

The documents do not discuss whether private housing contractors would bear any of the costs of the lead inspections, or how any repairs would be funded.

In most cases, Army post homes are now majority-owned by private real estate companies. Under their 50-year agreements with the Army, corporate landlords operating military housing agreed to control lead, asbestos, mold, and other toxic risks present in some homes, particularly historic ones.

FAMILIES, SENATORS PRESS FOR ANSWERS

The Army plans come as base commanders and housing contractors face a wave of complaints about potential home lead hazards, and a rush of military families seeking lead tests for their children.

Last week, the hospital at Fort Benning, where Reuters reported that at least 31 small children had tested high for lead exposure in recent years, began offering “walk-in” lead testing. Some concerned families are already being relocated; in other homes, maintenance workers are using painter’s tape to mark peeling paint spots that residents found contained lead by using store-bought testing kits.

Lead poisoning is preventable, and its prevalence in the United States has declined sharply in recent decades. Still, a 2016 Reuters investigation documented thousands of remaining exposure hotspots, mostly in civilian neighborhoods.

Last week, eight senators, including Republican Johnny Isakson of Georgia and Democrat Claire McCaskill of Missouri, pushed amendments to legislation to examine and address the military’s handling of lead exposure risks.

In coming weeks, Army officials plan to meet with lawmakers to address their concerns, the military documents show.

(Edited by Ronnie Greene and Michael Williams)