RAG has two independent failure modes: retrieval and generation. You can retrieve the right documents but generate a wrong answer (generation failure). You can retrieve the wrong documents but hallucinate a plausible-sounding answer (retrieval failure masked by confident generation). Without measuring both sides independently, you can't diagnose what's broken.

"It seems to work" is not a metric. You need numbers.