Best Practices for System Validation

Katie,

As an initial platform validation?

We ran a series of test scripts that were driven by role permissions and a “base user experience” in the system. We actually had a vendor go through establishing the steps to document and validate that the environment is acting as expected.

We documented each behavior that was expected, known concerns about the permission set, and then findings by running the script itself to verify that components were working as expected or not expected.

As a backup, we kept the test users that were used in the system so we dont lose the audit and have the document on record with us.

Hope this helps.

@dklinger Thanks for sharing this! 

Hey there - actually sales and marketing started with Docebo Learn and we are looking to expand the use to our quality system training. Right now I am reviewing the SOC 2 report for key points of design control, change management, data backup and recovery and e-signature. For updates for Cloud-systems where changes are pushed, I have a decision tree that will need to be revised as it is written for quarterly patching not as frequent as Docebo monthly routine. In sum, we will bounce off of the SOC audits, reviewing them annually and review monthly updates for new functions that impact quality system tasks. From there, decide level of risk and testing requirements. Certainly open to other ideas to make monthly reviews less resource consuming while still having a validated system from FDA/Notified Body perspective. And then of course, the API being developed to hook into our document management PLM system...that’s another validation to be determined.

@tancar thanks for this! What level of testing/validation are required by your team for the monthly releases? We do want to try and create some scripts or otherwise identify how we can make this easier for admins teams. Open to any and all recommendations. 

Hi KatieB

Classification of updates as (1) bugfix, (2) modified existing function, (3) new function or (4) deleted function (with ID#).

For new functions a link to a test summary abstract indicating
-  the testing performed by Docebo and
-  if there were any other functions impacted by the code change of the new function along with assurance there was no negative impact to the existing code (or if there was an impact what was it).

To clarify the latter point, a piece of existing code “A” gets altered in order to make new code “B” work. Does the code “A” change affect any other existing code? It’s the potential cascading impact to also be concerned about. 

@tancar 

Thanks for this suggestion. We implemented for Sales and other groups last year and now are in the process of implementing for our high compliance (FDA) related training. I love the process you outlined above and think we will adopt it. 

Hello Katie - I have asked our internal Training Director (Jamie) to reach out to our Customer Success Rep regarding this string as it is important to us. Hopefully we can see some motion forward on some this before we go live with our Quality Management System training deployment. Thank you for raising this topic.

I wanted to share that I continue to work with our Product, Development and Compliance teams to share feedback on client needs and identify ways for us to better support these efforts. My own background is in regulated industries and compliance so I understand the effort and weight of validating software!

Our team is working to add additional detail to release notes and our Compliance team are exploring some documentation updates that may be feasible.

Keep the ideas and feedback coming, very helpful!!!

Hi Katie,

My quality team has asked if Docebo can provide evidence of Part 11 e-signature compliance? As we need to validate the system, it would help tremendously to be able to that documentation. I will ask our account manager as well.

Also, they have asked several times if Docebo has a validation package, at a minimum a set of test scripts that can be used for initial system validation, up to a service where this work could be completed on our behalf.  - I have asked this question before and the answer has always been no.

As noted above, accurate and detailed release notes are also needed.

@telias that is what we are working towards so understanding what you need and what would be helpful is great insight. I can tell you our compliance team are working to add details to our SOC2 (ask your AM for a copy if you need it) and we are putting together some general templates for clients to create their own test scripts. Validation services is a good idea I will chat with my colleague about what that may look like!

Great question, @KatieBonilla!

With out recent implementation, we had Docebo migrate our content from our previous vendor. The best we could do at validating that everything made it in was to compare the number of rows in the CSV with the number presented in the bottom left corner of Course Management when filtered for a specific course type. Beyond that it required a lot of manually checking each course.

It would have been great to have reporting capabilities in New Reports that allowed us to get more data around our courses. In fact, this is still greatly needed post implementation. The existing reports are all user centric, but we also want to know about the courses themselves. We want be able to run a report on all courses to see data like title, length, for sale or not, and pretty much all of our additional fields like more detailed course type information, the product the course relates to, the curriculum level, and more. It would also be helpful to see the assigned instructor of a session. We’ve had to add an additional field at the session level and add and maintain our 60+ instructor roster so we can see who the instructor for a session is within the course management dashboard and not have to do multiple clicks within the course and sessions. The downside to our solution is additional fields can’t be sorted on the course management page, so it’s only partially helpful when we want to look at a specific instructor’s sessions clustered together.  Being able to report on course data would be such a huge win in terms of validation. When we make system-wide changes, it would be great to make the changes whether manually or by import, then export a report to make sure everything was updated properly. It would also be helpful when it comes to importing sessions and events that require system ID that can only be retrieved by looking in the URL or through API.

Following - as this is all great information.   Thank you @KatieBonilla for starting this conversation!