fleet/proposals/Fleetctl-trigger.md

Fleetctl trigger

Goal

As a user, I would like to trigger a set of async jobs using fleetctl. For example, I’d like to trigger a vuln scan, or an MDM dep sync.

The proposed solution to accomplish this goal enables a new CLI command: fleetctl trigger --name <NAME>.

Background

Currently, the Fleet server uses the schedule package to create sets of defined jobs that are run serially at defined intervals. The initial schedule interval must be specified at the point the schedule is instantiated via schedule.New. Optionally, schedule. WithConfigReloadInterval accepts a reload interval function. If specified, the reload interval function is periodically called and its return value becomes the new the schedule interval. This mechanism allows the schedule interval to be modified by user, for example, by changing the app config; however, there no mechanism currently to trigger async jobs on an ad hoc basis.

Proposal

New CLI command fleetctl trigger --name <NAME>

• Upon this command, the CLI client makes a request to a new authenticated endpoint (see below) to trigger an ad hoc run of the named schedule.

New schedule option WithTrigger

• This option adds a trigger channel on the schedule struct that will trigger an ad hoc run of the scheduled jobs.
• The trigger channel for each schedule is exposed via a new schedules map on the Service struct.

New endpoint GET /trigger?name={:name}

• The request handler first calls ds.Lock to check if the named schedule is locked.
• If the named schedule is unlocked, request handler sends a trigger signal on the schedule's trigger channel and the server responds with status 202 Accepted.
• If the named schedule is locked (presumably because the schedule is currently running), the server responds with status 409 Conflict and includes a message indicating the schedule is currently locked. It is up to the user to retry. To facilitate retries, the response message could be expanded to include additional status information, such as the expiration time of the current lock.

Schedule locks

• Currently, lock duration is based on the schedule interval.

  • Once an instance takes the lock, it will hold the lock for the duration of the interval, even after it has completed the jobs in the schedule.
  • For long-running jobs, the lock may expire before the current instance completes its run, meaning that it is currently possible for another instance to start an overlapping job.
  • If the lockholder instance is terminated or killed, locks are not released, which may frustrate a user's attempt to configure a shorter schedule interval before the lock held by the dead instance expires.

• Under this proposal, locks become more dynamic.

  • Current lockholder releases its lock once it finishes running the schedule.
  • Graceful shutdown process handles release of locks upon termination signals. Jobs are preemptable and an interrupt function must be specified for each job, e.g., schedule.New(...).WithJob("job_name", jobFn, interruptFn).
  • As a fallback for cases that can't be handled via graceful shutdown (e.g., SIGKILL), the expiration for a lock is initially set to a relatively short default duration (e.g., 5 minutes). The expiration is then periodically extended by the current instance so long as scheduled jobs are running. If the current instance dies without graceful shutdown, the lock will only be held by the dead instance for a short period.

Additional UX considerations

• What are some potential options that would be useful for the fleetctl trigger command?

  • Request the current status of the the named schedule without triggering a new run. For example, --status could provide the user with the running time of the schedule (this would require that we expand the locks table to include additional timestamp information, such as lock start time and lock release time).
  • Other useful options?

• What rules should determine when the interval ticker resets? Consider the following cases where s.scheduleInterval = 1*time.Hour:

  • The schedule is triggered at 55 minutes into the 1-hour interval and takes 1 minute to complete. When should the schedule run again? (a) after 4 minutes; (b) after 1 hour; (c) after 1 hour plus 4 minutes; (d) other
  • The schedule is triggered at 55 minutes into the 1-hour interval and takes 11 minutes to complete. When should the schedule run again? (a) immediately; (b) after 1 hour; (c) after 54 minutes; (d) other

• What should be logged?

  • Debug log if schedule runtime exceeds schedule interval to aid detection/troubleshooting of long-running jobs.
  • Other useful logs?