The first gravitational wave (GW) – gamma-ray burst (GRB) association, GW170817/GRB 170817A, had an offset in time, with the GRB trigger time delayed by ~1.7 s with respect to the merger time of the GW signal. We generally discuss the astrophysical origin of the delay time, Δt, of GW-GRB associations within the context of compact binary coalescence (CBC) – short GRB (sGRB) associations and GW burst – long GRB (lGRB) associations. In general, the delay time should include three terms, the time to launch a clean (relativistic) jet, Δtjet; the time for the jet to break out from the surrounding medium, Δtbo; and the time for the jet to reach the energy dissipation and GRB emission site, ΔtGRB. For CBC-sGRB associations, Δtjet and Δtbo are correlated, and the final delay can be from 10 ms to a few seconds. For GWB-lGRB associations, Δtjet and Δtbo are independent. The latter is at least ~10 s, so that Δt of these associations is at least this long. For certain jet launching mechanisms of lGRBs, Δt can be minutes or even hours long due to the extended engine waiting time to launch a jet. We discuss the cases of GW170817/GRB 170817A and GW150914/GW150914-GBM within this theoretical framework and suggest that the delay times of future GW/GRB associations will shed light into the jet launching mechanisms of GRBs.