Skip to main content
Answer

Are you background jobs running?

  • October 31, 2022
  • 16 replies
  • 199 views

lrnlab
Hero III
Forum|alt.badge.img+10

Since the move to the data lake and the new process of 4hr refreshes, it seems this process is hit and miss...Over the weekend ran some reports and all went fine...today, on 2 separate client site, both reports and other background jobs are completely stalled. Nothing running for hours. Have 2 outstanding tickets…

Question to the Community...anyone else experiencing these serious latency issues with reports and b/g jobs?

Best answer by lrnlab

Seems the issue was corrected yesterday afternoon after many cries and emails. Thank you to support for ensuring this was resolved as quickly as possible.. Still quite scary that the system can be bricked this hard. After 2 such occurrences within 2 weeks, I certainly hope this is the final fix. Still waiting to know the root cause. 

16 replies

billso
Helper II
Forum|alt.badge.img+1
  • Helper II
  • October 31, 2022

I haven’t seen any issues with my reports, but I’ll be watching. Sometimes a data lake fixes take time to propagate properly. 


Bfarkas
Hero III
Forum|alt.badge.img+6
  • Hero III
  • October 31, 2022

fortunately, have not seen yet *knocks on all the wood*
hope you’re sorted soon!


TeriV
Novice III
Forum|alt.badge.img
  • Novice III
  • October 31, 2022

I’m having latency issues too and I have been contacted by a fellow admin that they have received a “data refresh in progress” message.


Bfarkas
Hero III
Forum|alt.badge.img+6
  • Hero III
  • October 31, 2022

Are you by chance in the US-East AWS bucket that has been having issues and 500 errors? (or maybe that issue is causing your performance issues?)


lrnlab
Hero III
Forum|alt.badge.img+10
  • Author
  • Hero III
  • October 31, 2022

@Bfarkas we’re in Canada. Apparently the CA-1 server is also having issues but has not been reported on the status page


Bfarkas
Hero III
Forum|alt.badge.img+6
  • Hero III
  • October 31, 2022

ah, well hopefully the issue they found and resolved here helps there too!


lrnlab
Hero III
Forum|alt.badge.img+10
  • Author
  • Hero III
  • October 31, 2022

pretty sure they are not related as ours, I believe, is about the data lake..all jobs failing since 6:15 AM ET

 


Bfarkas
Hero III
Forum|alt.badge.img+6
  • Hero III
  • October 31, 2022

aka frozen lake.


lrnlab
Hero III
Forum|alt.badge.img+10
  • Author
  • Hero III
  • October 31, 2022

indeed


Annarose.Peterson
Hero III
Forum|alt.badge.img+7

Seems like it’s about the time of year for lakes to start freezing up.  🙃

I’m definitely noticing latency in my system today, but haven’t encountered any issues with data refresh over here, but definitely keeping an eye on it. 


jbridges
Novice III
  • Novice III
  • October 31, 2022

I have notice latency since the move as well.  I have several reports that are run nightly and sent out to various people.  They uses to run anywhere from 12am-4am central.  Now they are not running or being sent out until between 7-9am, sometimes not until mid-morning.  Our Project Managers rely on these to prepare for their calls with customers so it is a little frustrating with the latency and the inconsistent timing. 


Daniel
Hero III
Forum|alt.badge.img+3
  • Hero III
  • October 31, 2022

We’ve had some data refresh issues since the move, but thankfully everything seems to be running okay at the moment. We’re on the AU (Sydney) Region server.


lrnlab
Hero III
Forum|alt.badge.img+10
  • Author
  • Hero III
  • Answer
  • November 2, 2022

Seems the issue was corrected yesterday afternoon after many cries and emails. Thank you to support for ensuring this was resolved as quickly as possible.. Still quite scary that the system can be bricked this hard. After 2 such occurrences within 2 weeks, I certainly hope this is the final fix. Still waiting to know the root cause. 


billso
Helper II
Forum|alt.badge.img+1
  • Helper II
  • November 2, 2022

I'll try some reports today. I was seeing some latency issues yesterday. Thx for submitting the tickets! 


michelleoonk
Novice III
  • Novice III
  • November 2, 2022

We haven’t been able to run reports since Monday. The message says: Last update of reports' data: Data Refresh in Progress. We have an open ticket and have been waiting on a resolution. We’re in Texas. 


billso
Helper II
Forum|alt.badge.img+1
  • Helper II
  • November 2, 2022

We haven’t been able to run reports since Monday. The message says: Last update of reports' data: Data Refresh in Progress. We have an open ticket and have been waiting on a resolution. We’re in Texas. 

We’re in Massachusetts, and I just ran a custom report. The process took a little longer than usual, but it was done in less than a minute. Hopefully the issues are resolved soon.