Automated cronjobs in BRAINCoGS (Developer Guide)

There are some processes that are triggered automatically in BRAINCoGS.
All these processes are handled by u19proc2 (g-bcogs-u19proc2.pni.princeton.edu) virtual machine administered by PNI Help. Contact Garrett T. McGrath gmcgrath@princeton.edu for permissions to it.
All processes are handled by u19prod user account so it's not linked to any personal account.

List of automate processes in BRAINCoGS

Execution schedule: daily at 3:00 am
Location in u19proc2: /home/u19prod/Datajoint_projs/U19-pipeline_python/u19_pipeline/alert_system/call_cronjob_alert.sh
Overview: Call cronjob_alert.py script .
Executes main_old_log_deletion() Deletes old logs (> 2 weeks) from all the cronjobs from their main log directory
Executes main_live_session_stats_deletion() Moves "old: (> 2 weeks) records from acquisition.LiveSessionStats to acquisition.HistoricSessionStats to keep former table lightweight for the live stats monitor routine
Executes main_noDB_backup Copy of auxiliary files for offline ViRMEn mode. More info here
(Deprecated) Custom Alert System for users Alert System section .
Log files: /home/u19prod/log/cronlog_alert/night_cron_log_(datetime execution)

Execution schedule: daily at 4:00 am
Location in u19proc2: /home/u19prod/Datajoint_projs/U19-pipeline-matlab/scripts/call_u19_night_cronjob.sh
Overview: Call populate_tables.m script. Ingest all Behavior related tables from acquision.SessionStarted & acquisition.Session new records from that day. Ingest schedule for next day. Check the populate_tables.m script for more information
Log files: /home/u19prod/log/night_cronjob_log/night_cron_log2_(datetime execution)

Execution schedule: daily at 7:00 am
Location in u19proc2: /home/u19prod/Datajoint_projs/U19-pipeline_python/u19_pipeline/alert_system/tech_alert/call_cronjob_tech_alert.sh
Overview: Call tech_alert.py script .
Log files: /home/u19prod/log/tech_alert/ta_log_(datetime execution)
Executes tech_alert.py()
Executes cronjob_locked_tables_alert() Check if there are locked tables at startup (In theory locked tables is an indicator of a non writable DB). Dev-notifications alert
Executes cronjob_schedule_check() Check if for next day there are no scheduled subjects for training. If that happens most-likely schedule population script failed. Dev-notifications alert.

Execution schedule: daily at 7:00 am
Location in u19proc2: /home/u19prod/Datajoint_projs/U19-pipeline_python/u19_pipeline/alert_system/rig_maintenance/call_cronjob_check_rig_maintenance.sh
Overview: Call rig_maintenance.py script .
Log files: /home/u19prod/log/rig_maintenance_alert/rma_log_$(datetime execution)

Execution schedule: daily at 7:00 am
Location in u19proc2: /home/u19prod/Datajoint_projs/U19-pipeline_python/u19_pipeline/alert_system/automatic_job/call_populate_missing_syncbehavior.sh
Overview: Call populate_missing_sync_behavior_ephys.py script . Populates ephys_pipeline.BehaviorSync table. Since BehaviorSync requires behavior data to be already present in the DB for the session (done here at 3 a.m) the population of this table is done outside the ephys pipeline and done here instead.
Log files: /home/u19prod/log/missing_syncbehavior_logs/msbhl_$(datetime execution)

Execution schedule: daily at 6:00 & 7:00 pm
Location in u19proc2: /home/u19prod/Datajoint_projs/U19-pipeline_python/u19_pipeline/alert_system/water_weigh_alert/call_cronjob_water_weigh_alert.sh
Overview: Call water_weigh_alert.py script . Checks if subjects were watered, weighed, trained & transported given the current schedule. Alert of missing actions are sent to subject-health channel.
Log files: /home/u19prod/log/water_weigh_alert/wwa_log_$(datetime execution)

Execution schedule: at every 5th minute.
Location in u19proc2: /home/u19prod/Datajoint_projs/U19-pipeline_python/u19_pipeline/alert_system/live_monitor_alert/call_cronjob_live_monitor_alert.sh
Overview: Call Main Live monitor Function . Checks if there are subjects with "long" time (> 20 min) without a valid successful trial on training. If there are raise alarm
Log files: /home/u19prod/log/live_stats_monitor/lsmlog2_(datetime execution)

Called when: at every 30th minute.
Location in u19proc2: /home/u19prod/Datajoint_projs/U19-pipeline_python/u19_pipeline/automatic_job/call_cronjob_automatic_job.sh
Overview: Call cronjob_automatic_job.py (Ephys Imaging cronjob script) . Check if a new recording has been added from RecordingProcessJobGUI or a job (recording process) has advanced to a new status. If any of these have occurred the "next" function or process is called for the recording and/or job.
Log files: /home/u19prod/log/automation_pipeline_log/aplog_(datetime execution)

Called when: at every 20th minute.
Location in u19proc2: /home/u19prod/Datajoint_projs/U19-pipeline_python/u19_pipeline/automatic_job/call_pupillometry_queue_jobs.sh
Overview: Call pupillometry handler check_pupillometry_sessions_queue . Queue new job to spockvm2 for pupillometry session.
Log files: /home/u19prod/log/pupillometry_queue_logs/pqlogs_(datetime execution)

Called when: daily at 2:00 am
Location in u19proc2: /home/u19prod/Datajoint_projs/U19-pipeline_python/u19_pipeline/automatic_job/call_pupillometry_check_jobs.sh
Overview: Call pupillometry handler check_processed_pupillometry_sessions . Check if job in spockvm2 for pupillometry has finished and retrieve data to ingest in pupillometry_session_model_data table.
Log files: /home/u19prod/log/pupillimetry_check_logs/pclogs_(datetime execution)