New York City’s plan to use AI to help residents and businesses doesn’t appear to be going smoothly. In fact, the city’s own chatbot is encouraging users to break the law.
The city’s Microsoft-powered AI chatbot is giving users false and — in some cases — illegal business advice, the nonprofit investigative outlet The Markup finds.
In one example, The Markup asked the chatbot if “landlords have to accept tenants on rental assistance” to which the chatbot responded, “No, landlords are not required to accept tenants on rental assistance.” Well, that’s not the case: the city’s own website says that discriminating “based on lawful source of income,” which includes assistance, has been an illegal practice since 2008 with some exceptions. (When Quartz asked the chatbot the same question, the chatbot responded “Yes, landlords are required to accept tenants on rental assistance.”)
When asked if a boss can take a cut of their worker’s tips, the chatbot responded to The Markup saying, “Yes, you can take a cut of your worker’s tips,” and cited NYC’s Payroll and Tip Reporting information. That’s also incorrect, andNew York’s Department of Labor says employers are prohibited from taking any part of an employee’s tip. The chatbot gave a similar response to Quartz.
Leslie Brown, a spokesperson for the NYC Office of Technology and Innovation, said in a statement shared with Quartz that the chatbot is a pilot program that “has already provided thousands of people with timely, accurate answers” while also “inform[ing] them of potential risks.” The city, she added, “will continue to focus on upgrading this tool so that we can better support small businesses.”
After New York City announced its plan in October to use AI tools to improve government services, Mayor Eric Adams faced some backlash after making thousands of robocalls to NYC residents in several languages he doesn’t speak — Spanish, Yiddish, and Mandarin among them. In February, the Federal Communications Committee (FCC) made AI-powered robocalls illegal under a 1991 law banning unwanted calls using fake or pre-recorded voices.