Manually Triggering Overnight Scripts

This document is an overview of the overnight steps:  https://alta200036.sharepoint.com/:x:/r/sites/ALTADataMashup/_layouts/15/Doc.aspx?sourcedoc=%7B6E378A50-9046-4A82-A81C-7F4CFD464B92%7D&file=Nightly%20Steps.xlsx&action=default&mobileredirect=true

This page is a hidden Nightly Dashboard for all of the steps:   https://dw.alta.org/dashboard-nightly.php

The Impexium process of creating the Report Server copy and notifying ALTA via email is step one. 

• That notification usually comes in around 3:00 am, but could be later.

• Scripts check for that at 3:30, 4:00, and 4:30 am in case it's late. If it comes through after that, the script would have to be run manually, however, in most cases, if it hasn't run by then, it hasn't run.

• The RS processes should take about 30 minutes, but longer the later in the day, and if too long, will time out.

• A second set of scripts check for the RS side to be done, and then continue at 5, 6, and 7 am.

If the whole process hasn't run at all, meaning the Thaddeus landing page has two red messages about the data being old, then do the following.

1. Check the Nightly Dashboard, and see if the whole row is blank, including no notification.

   1. If all Blank, move to checking Email.

   2. If the first few are filled in but anything after that is blank, DO NOT CONTINUE. This requires specific troubleshooting by Chad.

2. Check the impxium.notifications@alta.org email box, or if you're getting those emails directly, did it come through.

   1. If Yes, but it was after 3:00 am, jump to Triggering the overnight.

   2. If No, then Impexium support needs to be contacted. They will see this weekday mornings normally around 8 am or so, and while we don't want to pester them, submitting a support request via Impexium as early as you notice it can't hurt.

Triggering the Full Overnight Script

This should ONLY be done when:

1. Everything in the row in today's Dashboard row is blank, it’s well past the times above, and REM has informed us whatever the issue was has been fixed and the RS is updated.

2. In that case, there is no notification email.

Some other notes:

• If it's past lunch, and the internal team can get by, it's better to wait for it to run overnight again. Mid day is okay, but can be weird.

• This will include the Registry change logs and push, which is ultimately what's the most dangerous aspect to have run twice a day over overlapping.

• If the email notification is late, but has come through, then the trigger will work.

• There is NOT a direct method of triggering the process partially.

Go to the Production ColdFusion Administrator here: http://172.16.1.2:8523/CFIDE/administrator/index.cfm

1. Under the first tab "Server Settings" switch to "Scheduled Tasks".

2. Go into editing “Nightly Trigger Manual” which is NOT scheduled.

   1. Change the dates to today and tomorrow, and the time to a few minutes from “now”.

   2. It should now show back on the list of all to “RUN ONCE” with the upcoming time.

3. Once that starts running, the Thaddeus Dashboard should start showing the steps, but the times will likely take much longer.

4. As soon as the RS steps are fully complete, then on the “Scheduled Tasks”, click ONCE to “run” the “Nightly Scripts Local Try 1” which will then pick up with the rest.

If Dashboard is Doubling Up

If when looking at the dashboard, you notice that there is a 2nd set of numbers moving a little behind the first set, this is a problem as the script is running twice. 

This should be rare, but it will be a problem when the processes overlap in the Registry aspect of things.

In this case, it's CRITICAL to stop and re-start the "ColdFusion Application Server" process so that the script doesn't continue into our local Registry steps.

1. RDP to Hostek “web1”

2. Open Services

3. Stop "ColdFusion Application Server" - Can take a bit, just wait.

4. Start "ColdFustion Application Server"

   • Can take a while, including a message that it couldn't start.

   • Wait and Refresh the Services list until it changes from "Starting" to "Running".

The MSSQL SPROC scripts will keep going since they were already running, but all next steps won't run.

If in this "middle" state, Chad will have to check things and probably re-start everything from scratch for a single run, but it depends where it was left.