o g@sddlmZddlmZmZmZmZddlZddl m Z ddl m Z ddl mZmZddlmZmZmZmZddlmZeeZGd d d ZGd d d e ZdS) ) getLogger)ListOptionalTupleUnionN)Fusion)AttentionMaskFormat) FusionUtils NumpyHelper) NodeProto TensorProtohelper numpy_helper) OnnxModelc@sNeZdZdZdefddZdefddZdd Zd d Z d e d e fddZ dS) AttentionMask: Fuse Attention subgraph into one Attention node. modelcCs2||_i|_i|_t||_tj|_||_ dSN) r mask_indice mask_castedr utilsr MaskIndexEnd mask_formatget_opset_version opset_version)selfrr`/var/www/visachat/venv/lib/python3.10/site-packages/onnxruntime/transformers/fusion_attention.py__init__s  zAttentionMask.__init__rcCs ||_dSr)r)rrrrrset_mask_format!s zAttentionMask.set_mask_formatcCs*||jvr||j|ksJ||j|<dSr)r)rmask mask_indexrrrset_mask_indice$s zAttentionMask.set_mask_indicecCs t|jdks Jtt|jSNr)lenrnextiter)rrrrget_first_mask)szAttentionMask.get_first_maskinputreturnc Cs`|jtjkrdS||jvr|j|S|j|r!|j|\}}n |j|\}}d}|r2||j |<|jtj kr?||j|<|S|j d}|j dkrlt jd|g|g|j ddd}|jt ddgt d d gn7d }|j|dur|jt j|tjdgdgd d t jd||g|g|j ddd}|jt d d g|j|||j|<|S)NTr! ReduceSum MaskReduceSuminputsoutputsnameaxeskeepdimsrort_const_1_reduce_sum_axesFr0 data_typedimsvalsraw)rrNoMaskrrfind_graph_inputrcast_graph_input_to_int32cast_input_to_int32rrcreate_node_namerr make_node attributeextendmake_attributeget_initializeradd_initializer make_tensorr INT64add_node)rr(casted input_name cast_node output_namemask_index_node axes_namerrr process_mask-sV          $    zAttentionMask.process_maskN) __name__ __module__ __qualname____doc__rrrrr"r'strrNrrrrrs rc)sxeZdZdZdddddgfdededed eed ed ed e e ffd d Z de de eeffddZde de eeffddZde fddZde fddZde de de fddZde de de e ffdd Zd!e d"e d#e fd$d%Zde de fd&d'Zd(e d)ee dfd*ee dfd+e dee dff d,d-Zd.e d/e d0e d(e d)ee dfd*ee dfdedee dffd1d2Z 3 3 3 3 3 3 dDd.e d/ee e dfd0ee e dfd(e d)ee dfd*ee dfdeded4e d5e de de de d6e d7e d8edee dff"d9d:Z 3 3 3 3 3  dEd;e d.e d/e d0e d(e d)e d*e dededeed?edee dff&d@dAZdBdCZZS)FFusionAttentionrNFSkipLayerNormalizationLayerNormalizationr hidden_size num_headsattention_maskuse_multi_head_attention!disable_multi_head_attention_biassearch_op_typesc sh|rdnd}t|||||_||_|r|nt||_||_||_d|_d|_ d|_ d|_ d|_ dS)NMultiHeadAttention AttentionT) superrrWrXrrYrZr[mask_filter_valuenum_heads_warninghidden_size_warning shape_infershape_infer_done) rrrWrXrYrZr[r\attention_op_name __class__rrrms  zFusionAttention.__init__concatr)cCst|jdkr;|j|jd}|j|jd}t|tjr;|jdkr;t|tjr;|jdkr;|d|d|dfS|j|j fS)aU Detect num_heads and hidden_size from Concat node in the following subgraph: SkipLayerNormalization or EmbedLayerNormalization / | MatMul Shape | | Add Gather(indices=0) | | | Unsqueeze | | | Concat (*, -1, 12, 64) | / Reshape | Transpose r2r) r$r(rget_constant_value isinstancenpndarraysizerXrW)rrhrX head_sizerrr)get_num_heads_and_hidden_size_from_concats     z9FusionAttention.get_num_heads_and_hidden_size_from_concat reshape_qcCsF|j|jd}|dur3|j|d}|dur"|jdkr"||St|jdd|j|j fSt |}t |dksJ|ddksJ|ddkrYtd |d |j|j fS|d}|d}||}|jdkr||jkr|j rtd |jd |d d|_ |j dkr||j kr|jrtd|j d |d d|_||fS)zDetect num_heads and hidden_size from a reshape node. Args: reshape_q (NodeProto): reshape node for Q Returns: Tuple[int, int]: num_heads and hidden_size r2NConcatz is not initializer.rirjrrkzq_shape_value=z7. Expected value are like [0, 0, num_heads, head_size].z--num_heads is z. Detected value is z. Using detected value.Fz--hidden_size is )rrCr( get_parentop_typerrloggerdebugrXrWr to_arrayr$rawarningrb)rrsq_shaperh q_shape_valuerXrqrWrrrget_num_heads_and_hidden_sizes2    $ z-FusionAttention.get_num_heads_and_hidden_sizeadd_qkcCs|js|jjdd|_d|_|jdurdS|j|jd}|j|jd}|dus/|dur:td|ddS||krItd|ddS|jdS) NT)updaterr2zone of the inputs of z is Nonezthe shape of two inputs of z is not same)rdrinfer_runtime_shapercget_edge_shaper(rwrx)rr~ input_0_shape input_1_shaperrrget_add_qk_strs  zFusionAttention.get_add_qk_strcsdttfdd|j}t|dkrSt|dks J|jd}tjdfddt|j Dg|dd }|j ||j |j |<S) N_maskcs|jdkSr#)output)node)mask_output_namerrsz0FusionAttention.reshape_add_qk..r2rrtcsg|]}qSrr).0_)r~rr sz2FusionAttention.reshape_add_qk..r.r/r0axis) listfilter nodes_to_addr$rr>r r?rangerXappendthis_graph_namenode_name_to_graph_name)rr~ concat_nodeconcat_node_nameconcat_add_qk_fp32r)r~rrreshape_add_qks     zFusionAttention.reshape_add_qkpast_kpast_vc Cs|jd}|jd}|ddd}|ddd}tjd|g|g|dgd}tjd|g|g|dgd}|j||j||j|j|<|j|j|<|jd} |dd ddd d } tjd||g| g| dd } |j| |j|j| <| S) zConcatenate past_k and past_v inputs to create past_kv input. Args: past_k (str): name of past K value past_v (str): name of past V value Returns: kv_output_name (str): name of past KV value Unsqueeze_5d.rr)r.r/r0r1rtz.valuez.kv_value_kvr) rr>replacer r?rrrr) rrrunsqueeze_k_nameunsqueeze_v_name k_5d_name v_5d_namek_5dv_5drkv_output_name concat_kvrrrrsD      zFusionAttention.concat_kvc Csd}|j|}|dur%tjtjdd|jjgdd|d}|j||j|j d}|j d}|d  d d }|d  d d }t j d||g|g|d } t j d||g|g|d } |j | |j | |j|j|<|j|j|<||fS) ahReshape past_k and past_v from 4D to 3D to use as inputs for multihead attention node. Args: past_k (str): name of past K value of shape 4D past_v (str): name of past V value of shape 4D Returns: k_3d (str): name of past K value of shape 3D v_3d (str): name of past V value of shape 3D kv_4d_to_3dNrint64dtyper0Reshape_3drrr-)rrCr from_arrayrnarrayrWrDrr>rr r?rrr) rrr new_dims_namenew_dimsreshape_k_namereshape_v_name k_3d_name v_3d_namek_3dv_3drrr reshape_kv4s8       zFusionAttention.reshape_kvpresent_k_namepresent_v_namekv_nodec Csd\}}|j|}|j|}|dur(tjtjddd|d}|j||j|dur@tjtjddd|d}|j||j|jd}|jd} t j d||g|g|dd } t j d||g|g| dd } |j | |j | |j|j |<|j|j | <dS) a?Split kv_node containing present KV values into separate present K and present V values. Args: present_k_name (str): name of output to store present K value in present_v_name (str): name of output to store present V value in kv_node (str): name of present KV values )index_0index_1Nrrrrr2Gatherr)rrCrrrnrrDrr>r r?rrr) rrrrk_indexv_indexk_dimv_dim gather_k_name gather_v_name present_k present_vrrrsplit_kves:      zFusionAttention.split_kvc Cs|ddd}|ddd}|jd}|jd}tjd|g|g|gdd}tjd|g|g|gdd}|j||j||j|j|<|j|j|<||fS)a}Transpose past_k and past_v from (B,N,P,H) to (B,P,N,H) Args: past_k (str): name of past K value of shape (B,N,P,H) past_v (str): name of past V value of shape (B,N,P,H) Returns: past_k_transpose (str): name of past K value of shape (B,P,N,H) past_v_transpose (str): name of past V value of shape (B,P,N,H) _transposedrr Transpose)rrjr2rk)r.r/r0perm) rrr>r r?rrrr) rrrpast_k_transposepast_v_transposetranspose_k_nametranspose_v_name transpose_k transpose_vrrr transpose_kvs.      zFusionAttention.transpose_kvq_addk_addv_add name_prefixcCs|j|jdp|j|jd}t|}t|}t|}|dur<|j|jdp6|j|jd} t| }|durW|j|jdpQ|j|jd} t| }tj|||fdd} dt|j } |d} |j | |j | g| d| S)Nr2rrrk _qkv_biasr0r6r7r8) rrCr(r ryrn zeros_likestackprodshaperDr6)rrrrrq_biasqbkbvbk_biasv_biasqkv_bias qkv_bias_dim bias_namerrrcreate_combined_qkv_biass($   $ $ z(FusionAttention.create_combined_qkv_biasq_matmulk_matmulv_matmulc$Cs"|jd}|jd|jdkr|jd|jdksJ|j|jd} |j|jd} |j|jd} t| } t| } t| }| j| jkrR| j|jksTJ| jd}tj| | |fdd |d|f}|d}|j || j |jd|jdg|d|d}t j d|jd|g|g|d }|j|j|<|g}|d }|j |tjdgdgd d |d }|j |tjdg|gd d |d}|j |tjdgd|gd d |d}|j |tjdgd|gd d |d}|j |tjdgdgd d |d}t j d||||g|g|jdd }|j|j|j<|d}t j d||||g|g|jdd }|j|j|j<|d}t j d||||g|g|jdd }|j|j|j<|} |}!|}"||||g|jr|dur|j|jdrodnd}#tt|j|j|#r||jd|#<|} |||j|j|j<|dur|j|jdrdnd}#tt|j|j|#r||jd|#<|}!|||j|j|j<|dur|j|jdrdnd}#tt|j|j|#r||jd|#<|}"|||j|j|j<|j|| |!|"fS)aCreate packed QKV MatMul node before MultiHeadAttention node. This is for the scenario where an Attention node should be created but cannot be created because past_key and past_value are separate inputs and not one concatenated input. Args: q_matmul (NodeProto): name of MatMul from Q path - (batch_size, sequence_length, hidden_size) k_matmul (NodeProto): name of MatMul from K path - (batch_size, sequence_length, hidden_size) v_matmul (NodeProto): name of MatMul from V path - (batch_size, sequence_length, hidden_size) q_add (NodeProto): name of Add from Q path k_add (NodeProto): name of Add from K path v_add (NodeProto): name of Add from V path num_heads (int): number of heads Returns: Union[NodeProto, None]: the node created or None if failed. MatMulrr2rrk _qkv_weightr_qkv_outr-_q_start_indexFr5_k_start_index_v_start_indexrj_end_of_qkv_index_qkv_last_axisr_q_outSlice_k_out_v_outN)rr>r(rCr ryrrnrreshaperDr6r r?rrr rFr0rAr[anyrr)$rrrrrrrrXmatmul_node_nameq_weightk_weightv_weightqwkwvwd qkv_weightqkv_weight_nameqkv_matmul_output qkv_matmul qkv_nodes q_slice_name k_slice_name v_slice_nameend_of_qkv_nameqkv_last_axis_nameq_slice_outputq_slicek_slice_outputk_slicev_slice_outputv_sliceq_outputk_outputv_outputinitializer_inputrrrcreate_packed_qkv_matmul_nodes ,    "                   z-FusionAttention.create_packed_qkv_matmul_noderkey_padding_maskrr packed_qkvc Cs|dksJ|dkr||dkrtd|d|dStdd|jjD}|jd}g}|rT||||||||\}}}||j d|j d|j dgndt |t urt |t ur|j ru||j d|j d|j dgnC||j d|j d|j dgn1t |t krt |t kr||vr||vr|j r||j d||gn||j d||gndS|j s|||||}||n|d| r| r|| | | | gn | s| r|| | g| g}|r|r|||gtjd |||d }d |_|jtd |g|S) a[Create a MultiHeadAttention node. Args: q_matmul (NodeProto): name of MatMul from Q path - (batch_size, sequence_length, hidden_size) k_matmul (NodeProto): name of MatMul from K path - (batch_size, sequence_length, hidden_size) or (batch_size, num_heads, past_sequence_length, head_size) v_matmul (NodeProto): name of MatMul from V path - (batch_size, sequence_length, hidden_size) or (batch_size, num_heads, past_sequence_length, head_size) q_add (NodeProto): name of Add from Q path k_add (NodeProto): name of Add from K path v_add (NodeProto): name of Add from V path num_heads (int): number of attention heads. If a model is pruned, it is the number of heads after pruning. hidden_size (int): hidden dimension. If a model is pruned, it is the hidden dimension after pruning. output (str): output name of MHA key_padding_mask (str): name of key padding mask add_qk (str): name of add after Q x K' past_k (str): name of past K value - (batch_size, num_heads, past_sequence_length, head_size) past_v (str): name of past V value - (batch_size, num_heads, past_sequence_length, head_size) present_k (str): name of present K value - (batch_size, num_heads, sequence_length, head_size) present_v (str): name of present V value - (batch_size, num_heads, sequence_length, head_size) packed_qkv (bool): whether to combine MatMuls from Q, K, V paths Note: This is for the scenario where an Attention node should be created but cannot be created because past_key and past_value are separate inputs and not one concatenated input. Returns: Union[NodeProto, None]: the node created or None if failed. rinput hidden size # is not a multiple of num of heads NcSg|]}|jqSrr)rrrrrrzCFusionAttention.create_multihead_attention_node..r^rr]r- com.microsoftrX)rwrxsetrgraphr(r>rrArtyper r[rSrrr r?domainr@rB)rrrrrrrrXrWrrr~rrrrrgraph_input_names mha_node_name mha_inputsrrrr mha_outputsmha_noderrrcreate_multihead_attention_nodefsX -  $$$    z/FusionAttention.create_multihead_attention_noder!r( add_qk_strscalecausalc6Csf|dksJ| dkr| |dkrtd| d|dSd}|dur-|dur-|dur-d}|j|jd}|j|jd}|j|jd}d\}}}|r|j|jdp`|j|jd}|j|jdpr|j|jd}|j|jdp|j|jd}|r|r|r|sdS|durt|jdd dSt|}t|}t|}|j|jksJ|jd}|jd}|jd}||kr|ksJJ| dkr| |krt d | d |d d} |j|jkrd} t |jdd}!t |jdd}"t |jdd}#d}$| r%t j |||fdd }%|!|"|#}$nt j |||fdd }%d|!}$|rt|}&t|}'t|}(t |&j})t |'j}*t |(j}+|)|*krd|!ksgJJ|+|#ksnJ| rt j |&|'|(fdd },|)|*|+}-nt j |&|'|(fdd },d|)}-|jd}.|js|j|.d|j||$g|%d|r|j|.d|j|-g|,d|jr| rtddS|jd|jd|jd|.dg}/|dur|/|tjd|/| g|.d}0nt| |.d|r|.dndg}/|dur|/|n|/d| o|}1|1r|| |}2|/|2| dur4|| }3|1s/|/d|/|3| g}4|rW|rW|dddddd}5|4|5||||5tjd|/|4|.d}0d|0_|0jtd|g|r||0jtddg|dur|0jtd|g| r|0jtd|!|"|#gg|jdur|0jtd t|jg|0S)!a+Create an Attention node. Args: mask_index (str): mask input q_matmul (NodeProto): MatMul node in fully connection for Q k_matmul (NodeProto): MatMul node in fully connection for K v_matmul (NodeProto): MatMul node in fully connection for V q_add (NodeProto): Add bias node in fully connection for Q k_add (NodeProto): Add bias node in fully connection for K v_add (NodeProto): Add bias node in fully connection for V num_heads (int): number of attention heads. If a model is pruned, it is the number of heads after pruning. hidden_size (int): hidden dimension. If a model is pruned, it is the hidden dimension after pruning. input (str): input name output (str): output name add_qk_str (str): name of Add node after Q x K' past_k (str): name of input for past K value past_v (str): name of input for past V value present_k (str): name of output to store present K value present_v (str): name of output to store present V value scale: scale before softmax causal: whether it is uni-directional mask. Returns: Union[NodeProto, None]: the node created or None if failed. rrrNTFr2)NNNzl is not an initializer. Please set do_constant_folding=True in torch.onnx.export to unblock attention fusionzInput hidden size (z3) is not same as weight matrix dimension of q,k,v (z:). Please provide a correct input hidden size or pass in 0rrkr^rrrzVMultiHeadAttention does not support relative_position_bias: cannot fuse the attention.r]r-rz.key_keyrrr rXunidirectionalr,qkv_hidden_sizesr`) rwrxrrCr(printr ryrrzrnr concatenaterr>rZrDr6rrr r?rrrrr$r@rArBr`float)6rr!rrrrrrrXrWr(rr+rrrrr,r-has_biasrrrrrrrrr qw_in_size kw_in_size vw_in_sizeis_qkv_diff_dims qw_out_size kw_out_size vw_out_sizeqkv_weight_dimrrrr q_bias_shape k_bias_shape v_bias_shaperrattention_node_nameattention_inputsattention_node past_existspast_kvrattention_outputs present_kvrrrcreate_attention_nodes . $$$                                z%FusionAttention.create_attention_nodec6Csz|}|jdkr|j|dd}|dur|}ndS|j|gdgd}d}|dur2|\}}} } } n|j|gdgd}|durI|\}}} } ndSg} t|jD]\} }||vr[qR||djdkreqR| |qRt| dkrsdS| d} |j|d d}|dur||jd}|durt|d kr|d}|jdkr|jd}n,dS|durt|d kr|jd}ndS|jdkr||}|D] }|jdkr|jd}q ||}|jd krt|jd kr|jd}||}dd|D}| ddkrdS|j| gdgd}|durt ddS|\}}}}d}d}d}gdgdfgdgdfgdgdfgdgdfgdgdfd}d}| D],\}} |j| | d| d}|duraqJ|d krhd!}|d"krod!}|d#krvd!}|durt d$dSd}!d}"d}#|r|\}}#}"}n|r|\}}!}#}"n|r|\}}}"n|\}}!}}"|j|"gdgd%}$|$dur|j|"gd&gd'}$|$durt d(dS|$d)}%|$d*}&|$d+}'|j|"gdgd}(|(dur |j|"gd,gd-}(|(dur t d.dS|(d*})|(d+}*d}+d},|r:|j |#gd/gdfgd0gdfgd1gd2fg|\}}+}nS|rp|j |#gd3gd2fgd0gdfg|\}}+}|!duro||!},|,durot d4|!dSn|rtn|j |!gd5gd6fgd7gd8fg|\}}+}|s|+durt d9dS|st|+dkr|+djd kr|j|+d\}}-|-d:kr|-|_|jd|kr|'jd|kr|*jd|kr|s|j|+d+jdnd}.|dur| n| }/||%\}0}1|0dks|1dkr t d;dS||.|'|*||&|)||0|1||/jd|, }2|2dur#dS|j|2|j|j|2j<|durs|jd}3d<|3}4|jd=|3tjd gtdd|0t|1|0gdd>}5|j t!"d?|/jd|5jg|4gd@|3|j|4|jd<|j#$|/| | g|j#$||j#$|j%s|$n|$dd+|j#$|j%s|(n|(dd+|j#$|j%s|n|dd+d!|_&dSdSdSdS)ANrVAddr)rHrrrr)NNrrr)rHEinsumrr)r2Nrrr2MulrjrUricSrr)rv)rchildrrrrrz(FusionAttention.fuse..rrk)rrrHr)r2rrNz&fuse_attention: failed to match v pathF)SoftmaxrHDivr)rrNr)rMrHrJr)rMWhererrN)rrrjr)rMrHrOr)rrrrj)rMrNr)rrr)path1path2path3path4path5rRTrSrTz'fuse_attention: failed to match qk path)rrrN)rNrrrHr)rrrrNz&fuse_attention: failed to match q pathr)rrrrHr)r2rrrNz&fuse_attention: failed to match k path)ExpandrEqual)rXrr)CastrWrrX)rrrr)rYrXrrz4fuse_attention: failed to verify shape inference of )rJSubrYrr)Nrr2rr)rJrZrr)Nrr2rz)fuse_attention: failed to match mask pathizmFailed to detect num_heads and hidden_size for Attention fusion. Please specify those parameters in argument.edge_modified_shape_modified_tensorr5rreshape_modified_)'rvr match_parentmatch_parent_path enumerater(rrr$countrwrxitemsmatch_parent_pathsrget_constant_inputr`rYrNr}rzrGrrrr0rDr rFrnrintrGr r?nodes_to_removerArZ prune_graph)6rnormalize_nodeinput_name_to_nodesoutput_name_to_node start_nodeadd_before_layernormr einsum_noder reshape_qkv transpose_qkv matmul_qkv other_inputs_ir( root_inputmul_before_layernorm mul_childrenlayernorm_nodechildrenrL parent_nodechildren_typesv_nodesadd_vmatmul_v is_distillis_distill_addis_no_mask_attentionqk_pathsqk_nodeskvr~ matmul_qkwhere_qkq_nodesrsadd_qmatmul_qk_nodesadd_kmatmul_k mask_nodesr+mul_valr!attention_last_node q_num_heads q_hidden_sizenew_node unique_indexnew_edge shape_tensorrrrfuses                                   $ 0            zFusionAttention.fuse)rrrrrrF)rrrrrNF)rOrPrQrRrrerrboolrrSrr rrrr}rrrrrrrrrr*r3rGr __classcell__rrrfrrThs( )71-'                  |       frT)loggingrtypingrrrrnumpyrn fusion_baserfusion_optionsr fusion_utilsr r onnxr r r r onnx_modelrrOrwrrTrrrrs    V