Gov. Rick Scott often says that no actual citizens have been removed from the voter rolls in his program to make sure noncitizens don’t have the chance to cast ballots.

“Not one person has been taken off the voter rolls that was a resident, a U.S. citizen who has the right to vote,” Scott said Tuesday in Miami.

But that might not be the case.

In two counties — Collier and Lee — at least nine people have been removed from the voter rolls under Scott’s program, and elections officials have no solid proof that those people are noncitizens. More could be purged soon.

It’s that lack of certainty that concerns Democrats, liberals and voting-rights groups, who have sued the state to stop the program. On Tuesday, the U.S. Department of Justice also filed suit.

Critics say they worry that the program will spook legitimate voters who are immigrants.

“This affects the immigrant community and the rumor mill is churning,” said Lida Rodriguez-Taseff, a Miami lawyer representing the Advancement Project and a coalition of other liberal-leaning groups opposed to the program.

“People are in fear,” she said. “This is complicated and threatening.”

But Scott said he’s trying to make sure ineligible voters aren’t fraudulently casting ballots and “diluting” lawful vote.

More than 100 noncitizens have been spotted on the rolls so far, officials say, and nearly half might have voted.

The numbers are small and isolated, in large part because Lee and Collier appear to be the only two major Florida counties that are continuing with the program of purging potential noncitizens if they fail to respond to the counties’ requests to proof citizenship.

The other major Florida counties stopped the process amid concerns with the accuracy of a list of 2,700 potential noncitizens furnished by the state. The list disproportionately contained the names of actual citizens legally entitled to vote and incidentally happened to target more minorities than non-Hispanic whites and Republicans.

Had the large counties continued with the program, the controversy and questions would only have grown, judging by what’s happening in the two Southwest Florida counties. The state’s program could have been massive. The state initially identified more than 180,000 potential noncitizens on the rolls.

Scott said the process would have been far less controversial and far easier had the U.S. Department of Homeland Security not “stonewalled” the state by refusing to give Florida access to a database that contains immigrant information. The state sued DHS on Monday. Without the federal database, Florida elections officials used a state motor-vehicle database that contained citizenship information that isn’t always up to date.

Even before the program was essentially halted, there were signs it wasn’t being applied evenly. Local supervisors had to come up with their own standards to define citizenship and identify noncitizens.

Even now, in response to a Miami Herald query about how many noncitizens are on the rolls, supervisors apply different standards to answer the question. As a result, the numbers are uncertain: Either 140 noncitizens have been identified on the rolls, or the number is actually about 100, according to a Herald survey of state and county elections officials. No one is keeping a comprehensive statewide list reporting who has been purged and why.