._run/ sm
R ,,'
B))%*:*:DA
R))%*:*:LQ
'
D[
w}}n, - <. . >. .
' .
.
sC ' (.(8 8v 8>x >W^^ >
r r c e Zd ZdZdddZdd Z dddd
d d eddd
ededz dee j e f fdZ ddd
d d eddd
edeee
ee
f f dZedd ZddZy)CBGenerateManagerau Continuous batching generation via paged attention.
Translates between the handler's text-level asyncio.Queue and CB's
token-level interface. Per-request: ``max_new_tokens``, ``eos_token_id``.
The CB manager is initialized lazily on the first request via
:meth:`ensure_initialized`, using that request's ``gen_config`` for shared
sampling params (temperature, top_p, do_sample).
.. todo:: Remove :meth:`init_cb` when CB supports per-request
generation config. At that point, ``gen_config`` can be passed directly
to ``add_request`` and the CB manager no longer needs a shared config.
Nc d | _ || _ y r% )r
_cb_config)r& cb_configs r r' zCBGenerateManager.__init__i s #r r5 r r r r6 c | j y|j || j | _ | j j y)at Initialize the CB manager on first call with the request's generation config.
.. todo:: Remove when CB supports per-request generation config.
Args:
model (`PreTrainedModel`): The loaded model (must support ``init_continuous_batching``).
gen_config (`GenerationConfig`): Generation config used for shared sampling params.
N)r continuous_batching_config)r init_continuous_batchingr5 r r s r r zCBGenerateManager.init_cbm sA 88 11 (T__ 2
r rK r r r r c
| j }| t d t j }t j
|d } |j | |d|j |j }t |d| j }
t | j ||
|
|
fd} |j ||
fS )zFStart streaming CB generation. Registers a per-request output handler.3CB manager not initialized. Call `init_cb()` first.r' T)r streamingmax_new_tokens eos_token_idr8 r c j | | j rj y y # t $ r-}j t t
| Y d }~y d }~ww xY wr% )r is_finishedr r r r" r) )r r r
text_queues r
_on_outputz8CBGenerateManager.generate_streaming.._on_output sX
< V$%%' LLN (
<%%l3q6&:;;
rG r r register_result_handler)r& r5 rK r r r r cbr r' r% rB r rA s @@r r z$CBGenerateManager.generate_streaming~ s XX
: TU U'')$+MMO
;' ^^ !%44#00 $
!KCNN dhh
ND*bmn < "":z:8##r c K | j }| t d |d }t | }t j } | j fd}
|j
||
|j |||j d|j d{ } | t d| | j } |j | d }
|
|| fS 7 8w)
zcRun non-streaming CB generation. Registers a handler that resolves an asyncio.Future on completion.Nr; r' c J j sj | y y r% )doner )r r s r
_on_resultz._on_result s ;;=!!&) !r F)r r= r< r> z1CB manager stopped before producing a result for Tr( ) r rC r r r r rE rD r= r> r r* )r& r5 rK r r r rF r' r- r rJ r rZ r r s @r r z(CBGenerateManager.generate_non_streaming s XX
: TU U;' N '')##% * "":z:
!%44#00
> !RS]R^ _` ` //
4 HY
-- s BCC9Cc p | j t d | j j j S )z*The CB scheduler (for testing/monitoring).zCB manager not initialized.)r rC batch_processor schedulerr s r rM zCBGenerateManager.scheduler s0 88 <= =xx''111r c X | j | j j dd y y )NT )blocktimeout)r r r s r r zCBGenerateManager.stop s% 88 HHMMaM 0 r r% )r6 ContinuousBatchingConfig | Noner )r6 r r )r r r r( r' r r r) r r r r r r rY r propertyrM r r r r r3 r3 Z s $0 $(($ ($ >($ ($
' ($
($ D[($
w}}j( )($T&. &. >&. &.
' &.
&.
sCc" #&.P 2 2 1r r3 c ^ e Zd ZdZ ddededdfdZdd d
ed efd Zdd
eded e fdZ
ddZ y)GenerationStatea' Shared generation state across all handlers.
Manages per-model :class:`GenerateManager` instances (each with its own
:class:`InferenceThread` so different models can run concurrently while
``torch.compile`` / CUDA graphs require same-model-same-thread) and a
single :class:`CBGenerateManager` for continuous batching.
Args:
continuous_batching (`bool`, *optional*, defaults to `False`):
Whether to use continuous batching with paged attention instead of
sequential ``model.generate()`` calls.
Ncontinuous_batchingcompiler6 rR c X || _ || _ || _ i | _ d | _ d | _ y r% )_continuous_batching_compiler5 _generate_managers _cb_manager _cb_model_id)r& rV rW r6 s r r' zGenerationState.__init__ s2 %8!
#>@ 59(,r r5 r modalityr6 c | j syt |d xr |t j k( }|s,t j |j j d |S )aW Check if continuous batching can be used for this model and modality.
Args:
model (`PreTrainedModel`): The loaded model.
modality (`Modality`): The detected model modality (LLM, VLM, etc.).
Returns:
`bool`: ``True`` if CB is enabled and the model supports it, ``False`` otherwise.
Fr9 zM does not support continuous batching. Falling back to sequential generation.)rY r$ r r logger warning_oncer r )r& r5 r^ cans r use_continuous_batchingz'GenerationState.use_continuous_batching s] ((e 78UX =U ??++, -9 9
r r` use_cbc Z |rv| j |k7 r-| j !| j j d| _ | j "t | j | _ || _ | j S || j
vrt
| j
|< | j
| S )af Return a per-model generation manager, lazily created on first request.
Args:
model_id (`str`): The model ID in ``'model_id@revision'`` format.
use_cb (`bool`): Whether to return a CB manager or a sequential one.
Returns:
`BaseGenerateManager`: Either a `GenerateManager` or `CBGenerateManager`.
N)r6 )r] r\ r r3 r5 r[ r )r& r` rd s r get_managerzGenerationState.get_manager s H,##/$$))+'+D$ '#4t#O $,!## # 422 20?0AD # #H -&&x00r c ` | j "| j j d| _ yy)z$Stop any active generation managers.N)r\ r r s r shutdownzGenerationState.shutdown" s- ' ! ! # #D (r )FFNFr ) r r r r( r r' r rc r) r rf rh r r r rU rU so %* 7; - ! - - 5 --> ( W[ (1C 1 1BU 1.$r rU c e Zd ZU dZdZedz ed< e Zee ed< ddde
fdZ d e d
dfd Z
ed d
d
e fd Zd e d
ee ddf fdZ dd e ddded
dfdZedee ded
ee fd Zy) BaseHandlera Shared logic for chat completion and responses handlers.
Provides model resolution, generation config building, and SSE formatting.
Generation is delegated to the shared :class:`GenerationState`.
Args:
model_manager (`ModelManager`):
Handles model loading, caching, and lifecycle.
generation_state (`GenerationState`):
Shared state managing per-model generation managers.
N_valid_params_class_unused_fields
model_managerr generation_statec || _ || _ y r% )rn ro )r& rn ro s r r' zBaseHandler.__init__9 s
+ 0 r bodyr6 c ddl m} t |j }| j 1|t | j dt z
}|r
|dd| || j z }|rt j d| yy) zMValidate request fields against the handler's params class and unused fields.r
HTTPExceptionN__mutable_keys__i z"Unexpected fields in the request: status_codedetailz,Ignoring unsupported fields in the request: ) fastapirt r keysrl rG rm r` ra )r& rq rt
input_keys
unexpectedunuseds r _validate_requestz BaseHandler._validate_requestA s )%
# # /#gd.F.FHZ\_\a&bbJ#>`ak`l|| j j k7 r% |dd| j j d| d | j j |d< | j j |d }| j j
| \ }}|||fS ) zfApply force_model, load model + processor.
Returns ``(model_id, model, processor)``.
r rs r5 i zServer is pinned to 'z'; requested 'z'.rv )ry rt rn force_modelrS process_model_nameload_model_and_processor)r& rq rt requestedr` r5 rK s r _resolve_modelzBaseHandler._resolve_modelU s
* ) ) 5 )I$d6H6H6T6T)T# # 3D4F4F4R4R3SSabkalln o ! . . : :DM%%88g G --FFxPy ))r model_generation_configr rd c B ddl m} |j d |d i t j |d }n7t j | }|j |j dk rd|_ |j d +t |d |_ t |d dk( rd|_
|j d t |d |_ |j d t |d | j j r|j d
|_ |rd|_ |S ) a' Build a GenerationConfig from shared params (temperature, top_p, seed, generation_config JSON).
Subclasses should call ``super()._build_generation_config(...)`` then apply
endpoint-specific params (``max_tokens``, ``max_output_tokens``, etc.).
Args:
body (`dict`):
The raw request body.
model_generation_config (`GenerationConfig`):
The model's default generation config (will be deep-copied).
use_cb (`bool`, *optional*, defaults to `False`):
Whether continuous batching is active. If ``True``, disables the model's
internal KV cache (CB manages its own paged cache).
Returns:
`GenerationConfig`: A new config with request-specific overrides applied.
r )r r i temperatureg Ftop_pr staticr ) transformersr rS r/ loadscopydeepcopyr= floatr do_sampler r ro rZ cache_implementation use_cache)r& rq r rd r r s r _build_generation_configz$BaseHandler._build_generation_configj s ( 2 88' ( 4 0 Y4::dCV>W3X Y $
.E F //7;L;[;[^b;b37!0 88M " .,1$}2E,F )T-()S0.3!+ 88G (&+DM&: # 88F ' 4< ( ) ).?.T.T.\5= 2 */ ' ! r messagesr^ c g }| D ] }|d g d}d|v r|d |d< d|v r|d |d< d|v rg n|j d xs g }t |t rd|dg}|D ]b }|d }|d v r|d j d|d d (|d
v rT|t j
t j fv r2|d }t |t r|d }|d j d
|d |dk( r_|t j k( rL|d } t | t r| j dd nd}
| d } |d j dd|
d| d |dk( rA|t j
t j fv r |d j d|d d d *|dk( s1|t j k( sF|d j d|d d d e |t j k( r dj d |d D |d< |j | |S )a= Convert OpenAI-format messages to the format expected by HF processors.
All modalities extract text. VLM additionally handles ``image_url`` and ``video_url``.
MULTIMODAL handles all of the above plus ``input_audio`` and ``audio_url``.
For LLMs, the content parts are collapsed into a plain text string.
Args:
messages (`list[dict]`): OpenAI-format chat messages.
modality (`Modality`): The model modality (LLM, VLM, or MULTIMODAL).
Returns:
`list[dict]`: Processor-compatible messages.
role)r contentr= tool_call_idr r )r0 r r0 )r
input_text output_text) image_url input_imager urlimage)r0 r input_audioformatwavdataaudioz data:audio/z;base64, video_urlvideo audio_url c 3 & K | ] }|d yw)r Nr )rA rr s r rD zABaseHandler.get_processor_inputs_from_messages.. s ,R1QvY,Rrt )
rS rT r) r r r r r r join) r r^ processor_inputsmessager[ raw_contentr content_typer r fmt audio_b64s r "get_processor_inputs_from_messagesz.BaseHandler.get_processor_inputs_from_messages sR + ,G %fo"=F w&'.|'<|$ ()0 )@~& !- 7"gkk)>T>ZXZK+s+(. D E &
d &v #HH9%,,fgfo-VW!%AAhS[S_S_aiatatRuFu!+.C!#t, !%j9%,,gc-JK!]2x8CVCV7V")-"8K>HVZ>[+//(E:afC +F 3I9%,,g TWSXX`aj`kFl-mn![0X(,,PXPcPcAd5d9%,,ggkFZ[`Fa-bc![0XATAT5T9%,,ggkFZ[`Fa-bc-
d2 8<<'$'HH,Ry@Q,R$Ry! # #F +W+ ,X r ri )r r r r( rl r0 __annotations__r rm r) rU r' r r~ staticmethodr r r r r rY r r r r r rk rk ) s
(,+ "uNCH$1%1 *1 Yd Yt Y G 6 G3 G G *4 *E#7HJt2t,u *, W\0!0!3E0!OS0! 0!d < T$Z < 8 < X\]aXb < < r rk r )Br( r r enumr/ r abcr r collections.abcr concurrent.futuresr r r typingr transformers.utilsr pydantic
tokenizersr r r
r r r
r :transformers.generation.continuous_batching.continuous_apir 4transformers.generation.continuous_batching.requestsr 5transformers.generation.continuous_batching.schedulerr rn r
get_loggerr r` X_REQUEST_IDEnumr r" r r+ rI r rM rV rY r\ r^ r) r0 r r r r r r r r r r3 rU rk r r r r s # $ % & eUO+
H % tyy P9 P ? &F;
B+< B B8D T ".t .T
T@Q .*(
(
VDx D3 D4 DNE E PA2 A2H !& &R>># >>BA
) A
H1+ 1DJ$ J$Zp p r