How LLMs Fail and Generalize in RTL Coding for Hardware Design?

About the Taxonomy

Our four-level error taxonomy evaluates LLM-generated RTL code based on successive stages of the EDA pipeline:

• L1 Syntactic: The source string is rejected by the HDL parser. No AST can be constructed.
• L2 Semantic: The source string parses into a valid AST but violates at least one static semantic constraint (e.g., detected during elaboration, linting, or synthesis).
• L3S Functional-Solvable: The synthesized model fails to meet the design specification, but the model has demonstrated the ability to solve the problem in at least one other rollout (addressable via inference-time scaling / best-of-N sampling).
• L3U Functional-Unsolvable: The synthesized model fails to meet the design specification, and the model cannot solve the problem in any rollout (requires fundamental model improvement).

Evaluations on the VerilogEval Human benchmark reveal a strict empirical ceiling, with frontier models plateauing at a 90.8% initial pass rate. The solvability taxonomy exposes that L3U (Unsolvable) errors dominate across all model families, revealing persistent knowledge gaps that inference-time scaling cannot address. Our analysis exposes a striking surface convergence gap: optimization drastically reduces syntax errors but concurrently increases functional testbench failures. Ultimately, register transfer level (RTL) coding capacity relies heavily upon pretraining knowledge. Integrating reward and policy modelling (i.e., GRPO) during the post-training phase amplifies existing competencies by teaching models to compile, while L3S errors (addressable via best-of-N sampling) coexist with L3U errors (requiring model improvement).