Skip to main content

Update as of Dec 20, 2022: 

As you may already know, we slowed down the rollout process of the new data refresh in order to add a few improvements to the update and make it more robust for our customers.

 

We have abided by the plan for all regions, with the exception of the US, where the update for a small portion of platforms is not yet completed. We have concluded the implementation of the data lake for those regions and expect to activate the update on the missing platforms by mid-January 2023.

 

Over the last several months, we recognize that you have been experiencing delays in updating and extracting data for reporting purposes. Behind the scenes we have been working on a solution that involves building a new “data lake” (database where the LMS data is stored for reporting purposes) to remedy the issues. Now, after the final testing stage, we are ready to roll it out to all of our customers' instances, but before we do, we want to ensure you have everything you need to know.  

 

FAQ: Your Data Lake Questions Asked and Answered

In anticipation of this release, we thought we would start an FAQ with some of the questions we think you will find helpful. Please use the comments to ask any questions and we will be sure to get the answers! 

 

  1. Let’s level set, what is a ‘data lake’?
    A. Fair question, let's start at the beginning.  A data lake is a central repository where all structured and unstructured data is stored. The Docebo Data Lake is the database where all LMS data is stored and used specifically for reporting purposes. 

 

  1. How is this update going to improve the exporting of reports?
    A. Many customers have been experiencing delays in data updates, the Data Lake has been designed to improve response time with increases in nightly data refreshes, and a decrease in refresh errors. In short, you will not experience the same delays in updating and accessing your data.

 

Pro tip: By scheduling the reports, you can save the wait altogether as the data will refresh and send.

 

  1. So, does this mean I am getting realtime reporting?
    A. Well not real time, but much closer and much faster. Your data will be refreshed when triggered by the export of a report. If the data is not validated (i.e., it has been longer than 4 hours since the last refresh,) a new refresh will occur. This refresh will take approximately 10 minutes to complete before you can export the report. Data refreshes can happen up to 12 times a day.

 

  1. How can I take this update to the next level?
    A. Glad you asked!  Here’s that pro tip again - Schedule your reports to export on a specific time and day to avoid any delays and to get the most updated view of your data. 

 

  1. When can I expect my platform will be running at full steam?
    A. We will be rolling out this update gradually starting with sandbox instances this week. All other updates will be by regions as follow
  • APAC (ap-southeast-2) - The week of October 10
  • APAC (ap-southeast-2) - The week of October 10
  • Canada (ca-central-1)  - The week of October 10
  • Europe (eu-west-2) - The week of October 17
  • Europe (eu-central-1) - The week of October 17
  • United States (us-east-1) - October 10 - Nov 18 (as per the above update, a small portion of these platforms are not yet updated but will be activated on any missing platforms by mid-January 2023)
  • Europe (eu-west-1) - October 10 - Nov 18

 

Not sure where you fall? You can find which region you are hosted under Advanced Settings > Platform Information. 

 

  1. Will it interrupt other activities?
    A. Absolutely not! The automatic data refresh in the data lake or the platform will have no impact on the activities of the learners or the collection of data based on system activities. But you will have to wait until the update is complete (approximately 10 minutes) to preview your reports or export any others. 

 

  1. What will this look like on my platform?
    A. Here’s a quick look at what you’ll see with the new update.

 

In the two images below, you will see the manual refresh icon removed.
Before the update:
LTX0frpBt0fbj3WSKgqd9vjM9-wlm0qGvx1O0k8R3gFBs3U3OvB5RAfLHkzMZwdpnSSowh0T8RkEnUXxKm-wsWcF71Gntxtesuvj7DqYxiOEhACZVnH22WhPTkQRNBcf6VZrZ_J20veSuNa8fw5qQGn3k-iL2ljvCuPTF3RG3ET05_5jrSR6nGTKhg

 

After the update:

ywQNAXj2YpDaNLo9RMjEYqGKsO0SxTkn9-zLCzBzCAx1yK74Up-HvjmAFaOU9uQyDKs7Z7UncAlVDxuIRKo-gWw02i5nXptMePV9XZ4Ubm9-aN2T9nd5Lrlhrn0Iet6KyhxH_uGkmZ8FClxka7TrbU6OB_uFkrgN1qiSxelqw45mg0Bk7P9MFYDB1g

 

The two images below represent before and after screenshots of your “New Reports” page with a new section showing the status of the data refresh. 

Before the update: 

3zUJZ1maY9Tz_fHHMZb-3oPjFZe4a23U5tq76EK_gGv3MxFasBxWMdlupF3bmIEmv5OIoEk3lgdr660QjLylGUXxruHukjpDD2FDQgM0xbB16qjtS-fq57stq182erQ_zMMiyEF9LxUeAzigVI23A-lxTQx5Q81wtSlYY_diXv5jpVX8dRmeBnEIxg

 

After the update:

I06QNpaE22YRpakXafAGR8t9QbJkHagZTWTtqAp9Ob96hVFX0CUfOvrHy0-raEBUTLqso6O5d3LY6XqgNYuEskdsqXpRDy132kxYe9foP4FNgGVGE7LS7xdlB9-2RqO5dugksV5_uJkdmm0Za7my5sBNuOxZ7DOlXKxT-Bk9lfYpyAygC2qQu0wwPQ

 

  1. Where can I learn more about the update?
    A. New knowledge base alert!  Creating & Managing Custom Reports includes information on how to work with custom reports and in the update’s context, a detailed account on how the refresh takes place, an illustration of scenarios on the validity of the data and instructions on how to schedule custom reports. You can also find more details in the October 2022 Release Course in Docebo University.

 

 

If you have any more questions about the update, don’t hesitate to enter them below and we’d be happy to follow up!

This is a great informative update, thanks for making it so clear what is happening! Looking forward to more regularly refreshed data!

How will these refreshes affect Automation App and integrations that are running into the system? Is there still a timing we need to wait for the data refresh for integration to successfully run. 

 

Please correct me if I'm wrong.. 

My understanding is no more daily data refresh will occur in system due to this new automated data refresh.

Our admins can run a report anytime at docebo and the data will be refreshed when triggered by the export of a report.

 

It’s close to that, when you have a report run it looks to see when the data was refreshed and if it was greater than 4 hours it runs a refresh and then the report.

@sam.amin Thanks!  I was just posting about this yesterday!

Hi @Baskaran venugopal ,

yes, you’re right in saying that no more daily data refresh will occur in system and that the data will be refreshed when triggered by the export of a report. Just keep in mind that a data refresh to be triggered need 2 conditions:

  • a report export is requested
  • the data are older than 4 hours 

when these 2 conditions are met, the data are refreshed. 

Will there still be a automated daily data refresh?  How will this affect MY TEAM reporting? We don’t always run custom reports so we still need the data to refresh automatically for our MY TEAM users.

I think this is very exciting! Thank you! We have experienced a lot of issues with reporting delays.

Some food for thought. If a user runs a report and the system needs to be refreshed, a 10 minute delay still sounds like a poor user experience. 

❓ Is it possible to schedule the data lake to refresh every 4 hours, instead of requiring a refresh to be triggered by a report export? 

How will this update affect customers who, due to their plan, don’t currently have automated data refreshes and cannot schedule reports?  Will the process for refreshing data change for them?

This is amazing. Thank you Thank you Thank you!

Your FAQ isn’t clear - My instance states our hosting location is:

Europe (Ireland)

Your status page also lists an EU London. Which is eu-west-1 and which is eu-west-2?

 

Please amend  for clarity as we currently don’t know when this change will occur for us. 

 

If a user is attempting to run a report and both conditions are met (triggers report and it’s been more than 4 hours) will this cause any unseen delay for another user in the system working on another report? For example, I’m running a new report on consumption of courses, which causes a refresh, and while the refresh is occurring a co-worker also works on a new report. Will they see that the refresh is already in progress? 

 

Also, does the domain that a user is in refresh across all domains in the LMS? 

Hi everybody and tnx for all the questions.

@ebosworth automated daily refresh will no longer be available, but the manager reports request will as well trigger a data refresh if the data are expired.

@steveninfinger data refresh is one for all the users, which means that each user will see the refresh triggered by other users, but the report requests remain available, are enqueued and all delivered once the refresh is finished. 

Thank you for this!!!

Can’t wait for this, seems quite excellent, might actually move all the old reports over to new finally!

Hi everybody and tnx for all the questions.

@ebosworth automated daily refresh will no longer be available, but the manager reports request will as well trigger a data refresh if the data are expired.

@steveninfinger data refresh is one for all the users, which means that each user will see the refresh triggered by other users, but the report requests remain available, are enqueued and all delivered once the refresh is finished. 

Any chance of having my question answered? Which Europe West is which?

Hi everybody and tnx for all the questions.

@ebosworth automated daily refresh will no longer be available, but the manager reports request will as well trigger a data refresh if the data are expired.

@steveninfinger data refresh is one for all the users, which means that each user will see the refresh triggered by other users, but the report requests remain available, are enqueued and all delivered once the refresh is finished. 

Any chance of having my question answered? Which Europe West is which?

Apologies for the delay here @MichaelPlace. I’ve asked @nicolo malinverno to confirm this for you.

Hi @MichaelPlace 

sorry for missing your question. 

Regarding the regions:

Europe (Ireland)  eu-west-1
Europe (London) eu-west-2
Europe (Frankfurt) eu-central-1

 

 

Great improvement but it does not appear to be working for me. My manual refresh has disappeared but the data is not refreshing as you describe. I ran a report first thing this morning and then made some changes before 10 am. I then wanted to refresh the data and run the report again and discovered the manual refresh was gone and was directed to this post. 

I then scheduled the report for 11 as that was the nearest hour by that point. 

The scheduled report ran but the data was not updated.

At the top of my reports screen it says: Last update of reports' data: 21/10/2022 10:00:00 (GMT +01:00)

Shouldn’t the scheduled report at 11 updated the data at 11? Why was this not done?

And now 7 hours later it still hasn’t refreshed?

This is really frustrating if running the report with updated data is part of what you need to do in a day and there is no way to manually refresh.

@sam.amin, @nicolo malinverno @erin.brisson  

 

Hi @aswartz 

and thank you for reporting your experience. 

We are working on some enhancements on the UX to make the system behaviour more clear in the interface. 

Keep in mind 3 main rules:

  • to update the data is needed a report export request
  • the data are updated only when they are older than 4 hours
  • the refresh does not take real time data, but the more fresh data available, that usually are between 1 and 2 hours old. 

this should explain the behaviour that you report:

  • the manual data refresh is not longer present, but the data refresh is run automatically when you export a report (and the data are older than 4 hours) 
  • the scheduled report of 11, did not run the data refresh since the data where not older than 4 hours
  • after 7 hours you need to run a report (or schedule it) to have the fresher data available 

available for additional support

 

 

 

 

Will there be changes to the Admin Menu? Meaning will we still have Reports and New Reports or will there only be one?

Will there be changes to the Admin Menu? Meaning will we still have Reports and New Reports or will there only be one?

Yup, they are both still there.

This is a great enhancement. Many thanks for getting this done. 

Hi @aswartz 

and thank you for reporting your experience. 

We are working on some enhancements on the UX to make the system behaviour more clear in the interface. 

Keep in mind 3 main rules:

  • to update the data is needed a report export request
  • the data are updated only when they are older than 4 hours
  • the refresh does not take real time data, but the more fresh data available, that usually are between 1 and 2 hours old. 

this should explain the behaviour that you report:

  • the manual data refresh is not longer present, but the data refresh is run automatically when you export a report (and the data are older than 4 hours) 
  • the scheduled report of 11, did not run the data refresh since the data where not older than 4 hours
  • after 7 hours you need to run a report (or schedule it) to have the fresher data available 

available for additional support

 

@nicolo malinverno I’m also facing the data refresh issue. I can’t see the manual data refresh button & I refreshed the scheduled report by click “Export” then immediately get response as Data refresh happening. Still, more than 4 hours data refresh is running.

 

 

 

@Baskaran venugopal we do not see issue on your platforms… I’m going to DM you to check it together. 

 

 

Reply


Terms & ConditionsCookie settings