Google is reportedly asking contractors who are evaluating Gemini’s responses to rate signals outside their area of expertise. According to the report, the Mountain View-based tech giant has removed the option to skip prompts, which was used by these contractors if they felt they did not know enough about a topic to rate a response. Considering artificial intelligence (AI) hallucinations are a major concern for chatbots, this reported development could lead to a decline in the quality of Gemini’s responses on highly technical topics.
Google reportedly isn’t letting contractors leave Gemini prompts
According to a TechCrunch report, Google has sent a new internal guideline to contractors working on Gemini. Claiming to have seen a memo sent by the tech giant, the publication claims that these contractors are now being asked to answer questions even if they do not have the knowledge to accurately assess the answers.
Google reportedly outsources the evaluation of Gemini’s responses to GlobalLogic, a Hitachi-owned company. Contractors working on Gemini are said to be tasked with reading technical signals and rating the AI’s responses based on a number of factors such as truthfulness and accuracy. These individuals who evaluate chatbots have expertise in specific subjects like coding, mathematics, medicine, and others.
Until now, contractors could reportedly omit certain signals if it was outside their domain. This ensured that only those qualified to understand and evaluate the technical responses generated by Gemini were doing so. This is a standard post-training practice for basic models and allows AI firms to ground their responses and reduce the incidence of hallucinations.
However, that changed when GlobalLogic reportedly announced new guidelines last week that contractors are no longer allowed to leave prompts unless the response is “completely missing information” or contains harmful content that could A special consent form may be required for the assessment.
According to the report, the new guidance states that contractors should not “skip prompts that require specialized domain knowledge” and instead, they should rate the parts of the prompt that they understand. He was also reportedly asked to include a note mentioning that he had no knowledge of the domain.
A contractor said in an internal communication, “I thought the purpose of leaving was to increase accuracy by giving it to someone better,” the publication claimed.
