Monitor, cancel, and troubleshoot Alphabet scheduler jobs

Track job execution history, view dashboard stats, cancel or retry executions, and export job configurations for backup and recovery.

Alphabet exposes a set of monitoring endpoints that give you a live view of scheduler health, per-job execution history, and time-series execution data suitable for charts and alerting. Admin-level endpoints let you pause the entire scheduler for maintenance, purge old logs, and export or restore job definitions.

Dashboard stats

GET /api/v1/scheduler/dashboard/stats

Returns a summary of scheduler activity. Use this as the data source for a status dashboard or health probe.

curl -X GET https://your-api/api/v1/scheduler/dashboard/stats \
  -H "Authorization: Bearer <your_access_token>"

The response includes:

Field	Description
totalJobs	Total number of jobs defined in the system.
enabledJobs	Number of jobs that are currently active and scheduled.
successRate	Percentage of recent executions that ended in Succeeded.
upcomingExecutions	Count of executions due to fire in the near term.
recentFailureCount	Number of failures recorded in the recent monitoring window.

The Hangfire dashboard at /hangfire provides the same health data through a browser UI, including live queue depths, worker utilisation, and a searchable execution log. Access requires an authenticated admin session.

---

Execution history

List executions for a job

GET /api/v1/scheduler/jobs/{jobId}/executions

Returns a paged list of execution records for the specified job. All query parameters are optional.

Parameter	Type	Description
pageNumber	int	Page number (default: 1).
pageSize	int	Items per page (default: 20).
status	string	Filter by execution status: Pending, Running, Succeeded, Failed, Cancelled.
fromDate	ISO 8601	Return only executions that started on or after this date-time.
toDate	ISO 8601	Return only executions that started on or before this date-time.

curl -X GET "https://your-api/api/v1/scheduler/jobs/3fa85f64-5717-4562-b3fc-2c963f66afa6/executions?status=Failed&pageSize=50" \
  -H "Authorization: Bearer <your_access_token>"

Get a single execution

GET /api/v1/scheduler/executions/{executionId}

Returns the detailed record for one execution, including output, duration, retry attempt number, and final status. Returns 404 Not Found if the ID does not exist.

curl -X GET https://your-api/api/v1/scheduler/executions/9c8b7a6f-0000-0000-0000-000000000099 \
  -H "Authorization: Bearer <your_access_token>"

Execution timeline

GET /api/v1/scheduler/executions/timeline?hours=24

Returns time-bucketed execution counts suitable for plotting a chart. Each bucket contains counts for Succeeded, Failed, and Running executions within that time window. The hours parameter defaults to 24 if omitted.

curl -X GET "https://your-api/api/v1/scheduler/executions/timeline?hours=48" \
  -H "Authorization: Bearer <your_access_token>"

---

Failed jobs

GET /api/v1/scheduler/jobs/failed?threshold=3

Returns jobs whose consecutive failure count meets or exceeds the supplied threshold. The default threshold is 3.

curl -X GET "https://your-api/api/v1/scheduler/jobs/failed?threshold=5" \
  -H "Authorization: Bearer <your_access_token>"

Use this endpoint to drive automated alerting. If the response list is non-empty, the jobs in it have failed on every recent attempt and are not self-recovering.

---

Managing running executions

Cancel an execution

POST /api/v1/scheduler/executions/{executionId}/cancel

Attempts to stop an execution that is currently Running or Pending. Returns 200 OK on success. Returns 409 Conflict when the execution has already completed or cannot be cancelled in its current state.

curl -X POST https://your-api/api/v1/scheduler/executions/9c8b7a6f-0000-0000-0000-000000000099/cancel \
  -H "Authorization: Bearer <your_access_token>"

Retry a failed execution

POST /api/v1/scheduler/executions/{executionId}/retry

Queues a new execution for the same job, using the selected failed execution as the retry parent. Returns 202 Accepted with the new execution ID.

curl -X POST https://your-api/api/v1/scheduler/executions/9c8b7a6f-0000-0000-0000-000000000099/retry \
  -H "Authorization: Bearer <your_access_token>"

The new execution ID is returned in the response body. Track it with GET /api/v1/scheduler/executions/{executionId}.

---

Admin operations

The following endpoints require the AdminOnly authorization policy. They are intended for operational use during maintenance windows and disaster recovery scenarios.

Calling POST /api/v1/scheduler/admin/pause-all stops every job in the system immediately. Ensure you have a plan to call resume-all before using this endpoint in production.

Pause all jobs

POST /api/v1/scheduler/admin/pause-all

Suspends every active scheduler job. Use before schema migrations, data patches, or infrastructure maintenance where job execution would cause conflicts.

curl -X POST https://your-api/api/v1/scheduler/admin/pause-all \
  -H "Authorization: Bearer <your_admin_token>"

Resume all jobs

POST /api/v1/scheduler/admin/resume-all

Restores all paused jobs and re-registers their schedules in Hangfire. Call this after maintenance is complete.

curl -X POST https://your-api/api/v1/scheduler/admin/resume-all \
  -H "Authorization: Bearer <your_admin_token>"

Clear old execution logs

DELETE /api/v1/scheduler/admin/clear-logs

Deletes execution records older than the specified number of days. Returns the count of deleted records.

curl -X DELETE https://your-api/api/v1/scheduler/admin/clear-logs \
  -H "Authorization: Bearer <your_admin_token>" \
  -H "Content-Type: application/json" \
  -d '{"olderThanDays": 60}'

Hangfire also expires completed job records automatically after Scheduler:Hangfire:JobExpirationDays (default: 30). The clear-logs endpoint operates on Alphabet’s own execution history records, giving you finer control over retention.

Export job definitions

GET /api/v1/scheduler/admin/export

Downloads all current job definitions as a JSON file (scheduler-jobs-export.json). The response content type is application/json.

curl -X GET https://your-api/api/v1/scheduler/admin/export \
  -H "Authorization: Bearer <your_admin_token>" \
  -o scheduler-jobs-export.json

Import job definitions

POST /api/v1/scheduler/admin/import

Creates jobs from a JSON payload exported by the export endpoint. Returns the count of successfully imported jobs.

curl -X POST https://your-api/api/v1/scheduler/admin/import \
  -H "Authorization: Bearer <your_admin_token>" \
  -H "Content-Type: application/json" \
  -d '{"jsonPayload": "<paste exported JSON string here>"}'

Import does not deduplicate. If you import a payload into a system that already contains jobs with the same names, you will end up with duplicate job definitions. Export and clear existing jobs first when restoring to a populated environment.

---

Troubleshooting

Jobs queue up but execute slowly

Your Scheduler:Hangfire:WorkerCount (currently 5) may be too low for your job volume. Increase it in appsettings.json and restart the application. Each worker processes one job at a time, so a value of 5 means at most five concurrent executions.

Scheduled jobs were missed after a restart

Hangfire stores its queue in the database. Any jobs that were enqueued at the time of the restart are recovered automatically when the application comes back up. You do not need to retrigger them manually.

A job is stuck in Running status

The execution may have been orphaned by a crash. If the execution is no longer actively running, cancel it with POST /api/v1/scheduler/executions/{executionId}/cancel and then retry with POST /api/v1/scheduler/executions/{executionId}/retry. Check your timeoutSeconds setting on the job — increasing it prevents future timeouts that leave jobs in an ambiguous state.

Disaster recovery

1. Export before migrations

   Before any schema migration or major upgrade, call GET /api/v1/scheduler/admin/export and store the resulting file in a safe location.

2. Back up the database

   Scheduler metadata and all execution history are stored in the primary application database. Include it in your regular backup schedule.

3. Restore job definitions

   After restoring a database or setting up a new environment, call POST /api/v1/scheduler/admin/import with the exported JSON payload to recreate all job definitions.