Overview
New York City launched the MyCity Chatbot in October 2023 to help business owners navigate regulations. In March 2024, The Markup and The City found the chatbot giving "dangerously wrong" information about local laws. It told landlords they did not need to accept Section 8 vouchers—contrary to city policy requiring landlords to accept lawful sources of income. The bot also misinformed users about worker pay, overtime rules and industry‑specific regulations. City officials said the chatbot is a beta product and included warnings that its answers may be incorrect, but critics argued that official branding gave its advice undue authority.
What Went Wrong
The chatbot’s training data and retrieval mechanisms failed to prioritize authoritative legal information, causing hallucinations and omissions. It presented incorrect legal advice with confidence, potentially encouraging illegal discrimination.
How It Was Fixed
NYC’s Office of Technology and Innovation acknowledged the inaccuracies and said it would "continue to focus on upgrading" the tool. Officials added warnings about the bot’s limitations and pledged to improve its retrieval process and include human review.