Worse Harmony Search Results

I agree, the search results are less than ideal. What is the logic behind the search. Do the same SEO rules we had for the global search still apply for Harmony?

My experience is that system related questions, Admin type answers have improved. We don’t use the global “for all users” Harmony because of concerns around the technical nature of our products and their use. It’s taken over a year of work for our internal AI build for this topic, so I’m looking forward to the MPC server connections to be able to integrate our own tool instead of using Docebo’s (sorry to  say.)

Correct. For admins, it is vastly better. For average users of our organization finding information, it is significantly worse. 

Yes I agree, we have been using Harmony internally for sometime.  At the start of the year it was working really well for us, we have 1 product with 100’s of pieces of content on it, Harmony was providing accurate answers and brining back the correct resources. Then it changed, we have noticed a decline in the answers that are coming through and 50% of the time it says it is not there to help with configuration. 

We have been going through the settings extensively and as we are not using the AI Copilot for administration duties, we have turned off this for our domain. It seems to have slightly improved the Harmony results but they are still not as good as they were. 

This is really disappointing for us as we were starting to rely on it as a vital tool. 

We launched Harmony to our field teams in January 2026 as a pilot supporting two products. It was working great until the update. We had a similar problem listed above when AI Copilot and Harmony merged. After working with our CSM and Docebo support, they had to implement some fixes. We’ve continued to test and provide feedback of issues we’ve had - sharing screenshots with them of the issues. I’d recommend anyone having issues do the same! My expertise is not in IT or AI so much of this was above my head. We have seen improvements lately. 

I’ve listed the issues and the solutions they deployed below.  

First Issue Sent - June 2026

The root cause was two compounding prompt issues in the administrator agent:
 
search_user_documents results were framed as "supplementary context" — lower priority than platform docs. When platform docs came back empty, the agent discarded the user-document results instead of answering from them.
 
The "general knowledge disabled" fallback instruction was a blanket rule: "if you don't find anything, tell the user you couldn't find anything." It fired even when retrieved content was present, because it didn't distinguish "retrieval returned nothing" from "platform docs returned nothing."

Solution
 
Three changes, all in the prompt/tool layer:
 
administrator.py — search instruction: search_user_documents is now explicitly marked as a primary, authoritative source for factual, definitional, and subject-matter questions. The agent is required to ground its answer on retrieved content when it is relevant, and is told that doing so is grounded retrieval — not general knowledge — so it is permitted even when general knowledge is disabled.
 
administrator.py — general knowledge disabled instruction: The refusal now fires only when neither retrieval tool returned anything relevant, not whenever platform docs alone are empty.
 
builder.py — tool description: The search_user_documents description was updated to reflect that it covers catalog and course/learning content the user can access, not just personal uploads — preventing the agent from misclassifying content questions as out of scope.

Second Issue Sent - July 2026

The issue was : Harmony sometimes found the right source, but then a separate AI “checker” wrongly decided the source was not relevant, so it answered with “I can’t find relevant resources.” Other times, that same checker let it through, so the user got a proper answer.
  
The fix does two things:

• Instead of asking the final AI checker once, it asks 3 times in parallel and keeps the documents if any one of those checks says “yes.” This makes the decision much more stable.
• After rewriting the question, it now merges the original results with the rewritten-query results instead of replacing them, so useful documents are no longer lost.