Skip to main content

Structured MCP Server evaluation from a live sandbox — 60% timeout rate on semantic search

  • April 30, 2026
  • 1 reply
  • 75 views
vegard.ofstaas
Influencer II

We've run a structured, hands-on evaluation of the Docebo MCP server beta against our Visma Learning Universe sandbox instance. Our platform serves product and partner training for Payroll, Visma Net, and Business NXT, primarily in Norwegian — so this is a real-world enterprise test with non-English content.

How we tested: We used Claude (Anthropic) in Cowork mode as the MCP client, which gave us a realistic end-to-end view of how the server performs when consumed by an actual AI assistant. We tested all four tools with domain-specific queries, cross-verified results against the Docebo UI, and ran security probes including prompt injection and SQL injection attempts.

What impressed us: The permission model is excellent — properly user-scoped, read-only, no admin data leakage. The semantic search returns genuinely useful step-by-step answers from our training material when it works. Cross-language search (English query returning Norwegian content) is a nice bonus.

What blocks us from going further:

  1. ask_learning_content timeout rate: 60% — 3 of 5 semantic queries timed out at 180s in our second test round. The queries that failed were core use cases for our learners ("Hvordan kjører jeg lønn i Payroll?"). This is the single biggest blocker.
  2. No pagination on get_my_learning_enrollments — our users have 200+ enrollments, producing 150K+ character responses that overflow LLM context windows.
  3. UTF-8 encoding issues in RAG chunks — Norwegian characters (åøæ) render as mojibake in retrieved content.

The full report (attached) includes a complete test appendix with every query, its result, and a cross-verification table against the sandbox UI. Happy to run additional tests or jump on a call if the team wants to dig into any of the findings.

Vegard Øfstaas Visma Software

      1 reply

      luca.latini
      Docebian
      • luca.latini
      • Docebian
      • April 30, 2026

      We've run a structured, hands-on evaluation of the Docebo MCP server beta against our Visma Learning Universe sandbox instance. Our platform serves product and partner training for Payroll, Visma Net, and Business NXT, primarily in Norwegian — so this is a real-world enterprise test with non-English content.

      How we tested: We used Claude (Anthropic) in Cowork mode as the MCP client, which gave us a realistic end-to-end view of how the server performs when consumed by an actual AI assistant. We tested all four tools with domain-specific queries, cross-verified results against the Docebo UI, and ran security probes including prompt injection and SQL injection attempts.

      What impressed us: The permission model is excellent — properly user-scoped, read-only, no admin data leakage. The semantic search returns genuinely useful step-by-step answers from our training material when it works. Cross-language search (English query returning Norwegian content) is a nice bonus.

      What blocks us from going further:

      1. ask_learning_content timeout rate: 60% — 3 of 5 semantic queries timed out at 180s in our second test round. The queries that failed were core use cases for our learners ("Hvordan kjører jeg lønn i Payroll?"). This is the single biggest blocker.
      2. No pagination on get_my_learning_enrollments — our users have 200+ enrollments, producing 150K+ character responses that overflow LLM context windows.
      3. UTF-8 encoding issues in RAG chunks — Norwegian characters (åøæ) render as mojibake in retrieved content.

      The full report (attached) includes a complete test appendix with every query, its result, and a cross-verification table against the sandbox UI. Happy to run additional tests or jump on a call if the team wants to dig into any of the findings.

      Vegard Øfstaas Visma Software

      Hi ​@vegard.ofstaas ,

      thanks for this, it's exactly the kind of real-world testing that helps us improve toward the general availability release!

      Here's my initial response. We'll dig into the logs and I will come back with a more detailed follow-up.

      1. ask_learning_content timeouts: This is not expected behavior and we haven't seen timeout rates like this before. It may be a transient issue in your sandbox environment, but we can't rule out other causes without a closer look. We'll investigate and get back to you.
      2. Pagination on get_my_learning_enrollments: This is a known limitation of the current implementation. The scenario you described (200+ enrollments) is exactly the kind of case we're working to handle better. It's on our radar.
      3. UTF-8 encoding: First report we've had on this. We'll dig in.

      Thanks again, the quality of your report is genuinely helpful.

        Terms & ConditionsAccessibility statement