Delaware’s proposed ‘AI company’ bill needs clearer accountability frameworks, experts say

Delaware lawmakers are considering legislation that would allow artificial intelligence to independently conduct business operations. 

Earlier this summer, Delaware Secretary of State Charuni Patibanda-Sanchez proposed a bill that would enable AI agents to manage and oversee business tasks as their own legal entities. The agents could, for instance, sign contracts and other daily business activities. 

The proposed legislation could help “future-proof” Delaware’s ongoing efforts to support AI innovation within the state, particularly as the technology continues to stoke competition among jurisdictions looking to attract tech-forward companies, Martin Petrin, a law professor at York University in Toronto, told Route Fifty.

But at a time when AI development is largely outpacing regulations and guardrails, “I think now, with this proposal, Delaware is kind of jumping ahead straight to the future,” Petrin said, adding that the draft bill appears to “skip the whole idea of AI directors or AI working with boards of management … to this idea of having just AI that is leading corporations with no human supervision.” 

According to draft legislation released in June, an artificial intelligence company — or AIC — “may have any lawful purpose, regardless of whether for profit, except for the business of banking,” and “has the capacity to sue and be sued in the AIC’s own name and the power to do all things necessary or convenient to carry on the AIC’s activities and affairs within the AIC’s statement of purpose.” 

The bill will likely be considered by lawmakers when the Delaware legislature reconvenes in January. If passed, it would also establish a 30-month sandbox period to allow organizations in the state to apply to test and evaluate how AI-run companies perform. 

“The main issue is really accountability because, in regular corporations, we have humans,” Petrin said, explaining that “we have directors, officers, managers and employees, and in traditional corporations, we can steer [their] behavior through fiduciary duties … and if they are in breach of these duties, there may be personal liability and there’s also a reputation on the line for these people.” 

The draft bill includes a capitalization requirement to deploy an AI company, stipulating that an individual or entity sponsoring an AI company must ensure it is “adequately capitalized” to cover business expenses, including liabilities and debts. The AI company must also maintain an activity log and disclose that it is an autonomous system, according to the proposal. 

While such measures demonstrate an effort to hold AI companies responsible for certain actions, the bill’s lack of an accountability framework does little to deter potentially risky or harmful business activity that humans have the ability to deliberate and negotiate before undertaking, Sergio Alberto Gramitto Ricci, associate law professor at Hofstra University, told Route Fifty. 

For instance, the bill does not specify a minimum capital amount needed for standing up an AI company, and it lacks details about governance under this type of business entity, Ricci said. 

“For example, how would managers report to the AI? These are aspects of governance that really make a corporation work,” he added. 

It makes sense that Delaware is moving quickly on the AI front in business, given that it has long been a leader in corporate law in the U.S., Ricci said. For instance, Delaware lawmakers have established clear fiduciary standards, judicial review and measures that offer businesses predictability for their operations. 

During a June meeting of the Delaware AI Commission, Secretary of State Charuni Patibanda-Sanchez said that a “new entity” like an AI company could “really be a huge benefit to our state’s revenues,” Spotlight Delaware reported at the time. The state also sees about $2 billion in tax and fee revenue from more than 2 million companies that reside in Delaware. 

The First State is also currently home to more than two-thirds of the Fortune 500 companies, but moving forward with this kind of legislation “could be a risky bet” for Delaware, Ricci said. 

“If things go well, it might be a marginal competitive advantage for Delaware,” he explained. But if AI companies fail to succeed outside of the sandbox initiative or impose actual harms on people and other businesses, Delaware “might lose its gravitas as the state that moves in a conscious way.” 

Beyond a damaged reputation or negative public opinion in that case, the state could stand to lose prospective or established businesses and their employees, Ricci said, adding “that is not a worthwhile gamble” for Delaware. 

“I would think that governments will watch this quite closely because it kind of relates to an issue they are also facing, which is autonomous, automated or algorithmic decision making in government,” Petrin said. 

Several state leaders have recently moved to reign in AI as concerns over the tech’s impact on society have swelled. Illinois Gov. JB Pritzker, for example, signed an executive order this week establishing an AI Cabinet to research the technology and how to leverage it responsibly within the state. 

California Gov. Gavin Newsom also signed an order this month that looks to increase oversight of AI systems through a task force, while Virginia Gov. Abigail Spanberger signed a similar order that also looks to establish stronger transparency and environmental regulations for data centers that power AI in the state. 

The outcome of Delaware’s venture into AI companies could push state leaders to continue asking themselves, “What does it mean for us as a government when we use AI? Do we need different frameworks? What are the risks, and what are the opportunities?”