Automated Cron Jobs in BRAIN CoGS (Developer Guide)

Some processes in BRAIN CoGS run automatically.
All of them run on the u19proc2 (g-bcogs-u19proc2.pni.princeton.edu) virtual machine, administered by PNI Help. Contact Garrett T. McGrath (gmcgrath@princeton.edu) for permissions.
All processes run under the u19prod user account, so they are not linked to any personal account.

List of automated processes in BRAIN CoGS

Execution schedule: daily at 3:00 am
Location in u19proc2: /home/u19prod/Datajoint_projs/U19-pipeline_python/u19_pipeline/alert_system/call_cronjob_alert.sh
Overview: Calls the cronjob_alert.py script, which:
- Executes main_old_log_deletion(): deletes logs older than 2 weeks from every cron job's main log directory.
- Executes main_live_session_stats_deletion(): moves records older than 2 weeks from acquisition.LiveSessionStats to acquisition.HistoricSessionStats, to keep the former table small for the live stats monitor routine.
- Executes main_noDB_backup: copies auxiliary files for offline ViRMEn mode (more info).
- (Deprecated) Runs the custom alerts for users (see the alert system section).
Log files: /home/u19prod/log/cronlog_alert/night_cron_log_(datetime execution)

Execution schedule: daily at 4:00 am
Location in u19proc2: /home/u19prod/Datajoint_projs/U19-pipeline-matlab/scripts/call_u19_night_cronjob.sh
Overview: Calls the populate_tables.m script. Ingests all behavior-related tables from acquisition.SessionStarted and acquisition.Session for records created that day, and ingests the schedule for the next day. See the populate_tables.m script for details.
Log files: /home/u19prod/log/night_cronjob_log/night_cron_log2_(datetime execution)

Execution schedule: daily at 7:00 am
Location in u19proc2: /home/u19prod/Datajoint_projs/U19-pipeline_python/u19_pipeline/alert_system/tech_alert/call_cronjob_tech_alert.sh
Overview: Calls the tech_alert.py script.
Log files: /home/u19prod/log/tech_alert/ta_log_(datetime execution)
Executes tech_alert.py()
Executes cronjob_locked_tables_alert(): checks for locked tables at startup (in theory, locked tables mean the DB is not writable). Sends a dev-notifications alert.
Executes cronjob_schedule_check(): checks whether any subjects are scheduled for training the next day. If none are, the schedule population script most likely failed. Sends a dev-notifications alert.

Execution schedule: daily at 7:00 am
Location in u19proc2: /home/u19prod/Datajoint_projs/U19-pipeline_python/u19_pipeline/alert_system/rig_maintenance/call_cronjob_check_rig_maintenance.sh
Overview: Calls the check_rig_maintenance.py script.
Log files: /home/u19prod/log/rig_maintenance_alert/rma_log_$(datetime execution)

Execution schedule: daily at 7:00 am
Location in u19proc2: /home/u19prod/Datajoint_projs/U19-pipeline_python/u19_pipeline/alert_system/automatic_job/call_populate_missing_syncbehavior.sh
Overview: Calls the populate_missing_syncbehavior_ephys.py script. Populates the ephys_pipeline.BehaviorSync table. Since BehaviorSync requires the session's behavior data to already be in the DB (ingested here at 4 a.m.), this table is populated here rather than in the ephys pipeline.
Log files: /home/u19prod/log/missing_syncbehavior_logs/msbhl_$(datetime execution)

Execution schedule: daily at 6:00 & 7:00 pm
Location in u19proc2: /home/u19prod/Datajoint_projs/U19-pipeline_python/u19_pipeline/alert_system/water_weigh_alert/call_cronjob_water_weigh_alert.sh
Overview: Calls the water_weigh_alert.py script. Checks whether subjects were watered, weighed, trained, and transported according to the current schedule. Alerts for missing actions are sent to the subject-health channel.
Log files: /home/u19prod/log/water_weigh_alert/wwa_log_$(datetime execution)

Execution schedule: every 5 minutes
Location in u19proc2: /home/u19prod/Datajoint_projs/U19-pipeline_python/u19_pipeline/alert_system/live_monitor_alert/call_cronjob_live_monitor_alert.sh
Overview: Calls the main live monitor function. Checks whether any subject has gone a "long" time (> 20 min) without a valid, successful trial during training, and if so, raises an alarm.
Log files: /home/u19prod/log/live_stats_monitor/lsmlog2_(datetime execution)

Execution schedule: every 30 minutes
Location in u19proc2: /home/u19prod/Datajoint_projs/U19-pipeline_python/u19_pipeline/automatic_job/call_cronjob_automatic_job.sh
Overview: Calls cronjob_automatic_job.py (the ephys/imaging cron job script). Checks whether a new recording has been added from the RecordingProcessJobGUI or whether a job (recording process) has advanced to a new status. If so, the "next" function or process is called for the recording and/or job.
Log files: /home/u19prod/log/automation_pipeline_log/aplog_(datetime execution)

Execution schedule: every 20 minutes
Location in u19proc2: /home/u19prod/Datajoint_projs/U19-pipeline_python/u19_pipeline/automatic_job/call_pupillometry_queue_jobs.sh
Overview: Calls check_pupillometry_sessions_queue in the pupillometry handler. Queues a new job on spockvm2 for the pupillometry session.
Log files: /home/u19prod/log/pupillometry_queue_logs/pqlogs_(datetime execution)

Execution schedule: daily at 2:00 am
Location in u19proc2: /home/u19prod/Datajoint_projs/U19-pipeline_python/u19_pipeline/automatic_job/call_pupillometry_check_jobs.sh
Overview: Calls check_processed_pupillometry_sessions in the pupillometry handler. Checks whether a pupillometry job on spockvm2 has finished and retrieves the data to ingest into the pupillometry_session_model_data table.
Log files: /home/u19prod/log/pupillometry_check_logs/pclogs_(datetime execution)