Prefill, decode and schedulingLesson 2 of 5Free preview
First token is not the whole answer
Distinguish time to first token, inter-token latency and completion time.
Prefill, decode and schedulingLesson 2 of 5Free preview
Distinguish time to first token, inter-token latency and completion time.