Your code parses JSON from an LLM, and about 2% of the answers fail to parse. What is the most robust approach?
Options
- Set the temperature to 0, which guarantees valid JSON
- Use the provider's structured output (JSON schema) mode if there is one, validate every answer against the schema in code, and retry with the error message when it fails (Correct option)
- Clean the text with regular expressions and hope
- Ask the model more politely to return JSON
Answer
Use the provider's structured output (JSON schema) mode if there is one, validate every answer against the schema in code, and retry with the error message when it fails
Why
Low temperature makes output more consistent, but it does not guarantee valid JSON.
Many providers can constrain the output to a schema. Even then, validate in code: the data can be valid JSON and still wrong, such as an empty field or a number out of range.
On a failure, a retry that includes the validation error usually fixes it, with a limit on the attempts.
Remember
Constrain, validate, retry with a limit.