What are the biggest mistakes with evaluation in llama.cpp?

[jessongaming]

I have a question about llama.cpp that may be useful for other people working with it too.

I am particularly interested in evaluation and real-world use. I would like to compare approaches that people have actually used rather than just theoretical suggestions.

What would you tell someone trying this for the first time?



Photo by

Please, Log in or Register to view URLs content!

on

Please, Log in or Register to view URLs content!

 

[charlieexplore]

The point about evaluation and llama.cpp is interesting. In practice I would keep the first test simple, record what worked, and only add extra complexity if it solves a specific limitation. If you share the exact setup, it would be easier to narrow down which part matters most.

 

[aria_zj7]

I think evaluation and llama.cpp is the key part of this topic. A practical approach is to define the outcome you want, test it with a small example, and then check quality, speed and reliability separately. That should make it easier to tell whether the change is actually helping.

 

[lucia_2nql]

For What are the biggest mistakes with evaluation in llama.cpp?, I would separate the problem into setup, output quality and repeatability. evaluation and llama.cpp fits into that because it can change the result quite a lot depending on the workflow. That should make it easier to tell whether the change is actually helping.