o gc@sddlmZddlmZmZmZmZmZddlm Z ddl m Z ddl m Z mZmZddlmZeeZGddde ZGd d d eZd S) ) getLogger)DictListOptionalTupleUnion)Fusion) FusionUtils) NodeProto TensorProtohelper) OnnxModelcs(eZdZdZd2dedeffdd Zdeded e eefffd d Z d ed e ee efde de fddZddZddZddZddZddZdede eed efffddZ  d3ded ed ed!ed"ed efd#eef d$d%Zd&d'Zd(d)Z d4d*d+Zd,d-Zd.d/Zd0d1ZZS)5FusionEmbedLayerNoMaskz Fuse embedding layer into one node (EmbedLayerNormalization). It supports the following model types: BERT, DistilBert, ALBert. no maskmodel descriptioncs<t|dddg|t||_d|_d|_d|_d|_dS)NEmbedLayerNormalizationLayerNormalizationSkipLayerNormalizationF)super__init__r utils shape_infershape_infer_done attention embed_node)selfrr __class__a/var/www/visachat/venv/lib/python3.10/site-packages/onnxruntime/transformers/fusion_embedlayer.pyrs  zFusionEmbedLayerNoMask.__init__addreturnNcCsP|j|dgdg}|durdS|j|dgdg}|dur dS|d|dfS)NGatherr)rmatch_parent_path)rr! gather_0_path gather_1_pathrrr match_two_gather&sz'FusionEmbedLayerNoMask.match_two_gather layernorminput_name_to_nodesis_distil_bertc Cs|jj|d|dd|_|jdurdS|jd|vrdS||jd}tdd|D}|gd kr_|D]+}|jd kr^|j|gd gd }|dur^|d jd|jdkr^|d|_dSq3t |dkr|djdkr|djd|vr||djd}t |dkr|djdkr|djd|vr||djd} | D] }|jdkr||_dSqtdd| D}|r|gdkr|gdkr|gdkrt ddSdS|gdkr|gd krt ddSdS)aCheck that LayerNormalization has a child of Attention node or subgraph like Attention. Args: layernorm (NodeProto): LayerNormalization node input_name_to_nodes (Dict[str, List[NodeProto]]): map from input name to nodes is_distil_bert (bool): whether it is DistilBert or not Returns: bool: whether there is Attention node or subgraph like Attention AttentionF) recursiveNTrcSg|]}|jqSrop_type.0childrrr KzCFusionEmbedLayerNoMask.check_attention_subgraph..)MatMulr6r6rr)Addr6MultiHeadAttentionr6)NNrrr$r6r7cSr.rr/r1rrr r4hr5)r6r6r6Shaper)r7r6r6r6r;r;)r7r6r6r6r;z Equal(B=?) -- Not -- Cast(to=6) -- CumSum(axis=1) -- Mul -- Cast(to=7) -- Add(B=1) -- Cast(to=7)* --> Gather | ^ V | +------------------------------+ Roberta new pattern from transformers v4.9: (input_ids) --> Equal(B=?) -- Not -- Cast(to=6) -- CumSum(axis=1) -- Add(B=0) -- Mul -- Cast(to=7) -- Add(B=1) --> Gather | ^ V | +-------------------------------------------+ start_node = position_embedding_gather start_index = 1 # match optional Cast node. parent = self.model.get_parent(start_node, start_index, output_name_to_node) if parent is None: return if parent.op_type == "Cast": if OnnxModel.get_node_attribute(parent, "to") != 7: return start_node = parent start_index = 0 i, path, return_indices = self.model.match_parent_paths( start_node, [ (['Add', 'Cast', 'Mul', 'CumSum', 'Cast', 'Not', 'Equal'], [start_index, 0, 0, 0, 0, 0, 0]), (['Add', 'Cast', 'Mul', 'Add', 'CumSum', 'Cast', 'Not', 'Equal'], [start_index, 0, 0, 0, 0, 0, 0, 0])], output_name_to_node) if path is not None: # constant input of Add shall be 1. i, value = self.model.get_constant_input(path[0]) if value != 1: return False _, self.padding_word_id = self.model.get_constant_input(path[-1]) return input_ids == path[-1].input[0] FrrrTrUrVrrr match_position_embedding_robertas-z7FusionEmbedLayerNoMask.match_position_embedding_robertac Cs|j|ddgddg|}|durdS|\}}|j|jd}|durTt|jdkrT|jddkrT|j|ddgrT|j|ddgrTt|jd ksV|j|d dgsVdS|j}|d krjt |d dgsidSn |j|ddgsudS|j |d|} | durdS| j d kr|j| ddsdS|j | d|} n| } | dus| j d krdS|j| ddsdS|j | d|} | dus| j dkrdS|| jdkS)a  Match position embedding path from input_ids to Gather for BERT. BERT Embedding Layer Pattern: (input_ids) / / Shape / | / Gather (indices=1) / | / Add (optional, B=0) / | Gather (segment_ids) Unsqueeze (axes=0) \ | | \ Gather Slice (data[1,512], starts=0, ends=*, axes=1, steps=1) \ / | Add Gather \ / Add | LayerNormalization SlicerNr$r:NFr axesr7r#r;) rr%get_constant_valuer?rArXrrSget_opset_versionr check_node_attribute get_parentr0) rrTrUrVpathslice unsqueeze slice_weight opset_versionrFgatherrXrrr match_position_embedding_bertsT   z4FusionEmbedLayerNoMask.match_position_embedding_bertcCs(||||r dS||||rdSdS)NTF)rpr^r_rrr match_position_embedding:s z/FusionEmbedLayerNoMask.match_position_embeddingc Cs|jd}|r |jdnd}|jd}|js!|jjdd|_d|_|jdurs|j|}|j|}|r6|s8Jt|dkrLt|dkrL|d|dksYtd|d|dS|rs|j ||sstd |d |j|dS|j |jd } | dust| j dkrtd dS|j |jd } | dust| j dks| j d| j dkrtd dS|r|j |jd } | dust| j dks| j d| j dkrtddS| j d | j d krt d|jd d| j d d|jd d| j d |rU| j d | j d kr-t d|jd d| j d d|jd d| j d | j d | j d krUt d|jd d| j d d|jd d| j d dS)zXSanity check of embedding weights, and match hidden_size of weights and shape of inputs.r$NT)updater:z^Cannot fuse EmbedLayerNormalization: input_ids and position_ids not matched in 2nd dimension: z vs FzYCannot fuse EmbedLayerNormalization: input_ids and segment_ids does not have same shape: z != rzICannot fuse EmbedLayerNormalization: word embedding table is not expectedzMCannot fuse EmbedLayerNormalization: position embedding table is not expectedzLCannot fuse EmbedLayerNormalization: segment embedding table is not expectedzword_embedding_table (z) size z <= position_embedding_table (z <= segment_embedding_table (zposition_embedding_table () r?rrinfer_runtime_shaperget_edge_shaperArBinfo compare_shaperfrXwarning) rword_embedding_gathersegment_embedding_gatherrTrU segment_ids position_idsinput_ids_shapeposition_ids_shapeword_embedding_tableposition_embedding_tablesegment_embedding_tablerrr check_embeddingHsj           222z&FusionEmbedLayerNoMask.check_embedding input_namecCsdd}|j|}|dur&|jjjtjkr |j|\}}||fS|}||fS|j|\}}||fS)aCast a graph input or node input to int32. Args: input_name (str): name of graph input or node input Returns: A tuple of casted input name and the cast node. int32_output (str): If input is int32, it is the input name, Otherwise it is output name of Cast node. input_cast_node (Union[None, NodeProto]): Cast node. It could be None if input is int32. N) rfind_graph_inputtype tensor_type elem_typer INT32rcast_input_to_int32)rrinput_cast_node graph_input int32_outputrrr cast_to_int32s z$FusionEmbedLayerNoMask.cast_to_int32FrUrxrTryr{c Csg} ||\}} |jd} |jdkr|jd} |jd} n |jd} |jd} d}|durL||jd\}} |||jd|jd|jd| | g}n|d|jd|jdd| | g}|durp|d||\}} ||| d | d g}|r|dur|n| d }||tjd||| d }d |_|j D]}|j dkr|j |gqt |j dkr|j t ddg| || D] }|j|j|j <q|j | ||_|S)agCreate an EmbedLayerNormalization node. Note that segment embedding is optional. Args: input_ids (str): input_ids for word embeddings layernorm (NodeProto): LayerNormalization or SkipLayerNormalization node. word_embedding_gather (NodeProto): the Gather node for word embedding position_embedding_gather (NodeProto): the Gather node for position embedding segment_embedding_gather (Union[None, NodeProto]): the Gather node for segment embedding, or None. Returns: NodeProto: the EmbedLayerNormalization node created. rrr$r:rbNr_output_dummy_mask_index_embedding_sum)outputsnamez com.microsoftepsilong-q=)rrcreate_node_namer0r?appendr make_nodedomain attributerextendrAmake_attributethis_graph_namenode_name_to_graph_name nodes_to_addr)rrUr)rxrTryr{embedding_sum_outputembedding_sum_namerrY node_namegammabetaembed_node_inputsrzembed_node_outputsrrattrFrrr create_fused_nodesl              z(FusionEmbedLayerNoMask.create_fused_nodecCs$|j|jd|jdd|_dS)NrT)rreplace_input_of_all_nodesr= prune_graph)rr)rrrr finish_fusion s z$FusionEmbedLayerNoMask.finish_fusioncCs*|jdkot|jdkot|jddkS)Nrrbr)r0rAr=)rrFrrr "is_skip_layer_norm_with_sum_outputs*z9FusionEmbedLayerNoMask.is_skip_layer_norm_with_sum_outputc Csx||}|dur dS|\}}|jd} |jd} |j||dds#dS||d|s,dS|jdkrP||} d} |} | rA|jdnd}|duoN|j|du}n@|} |jdkrYdnd} t |j| krg|j| nd}|duot|j|du}|o||vot ||dk}|duo|jdkp|p|} |j | ||||| | |r|ndd}| rd | j| <|s|j ||jd | ||d S) NFr$r+rrbr7r)rr_no_use__to_be_removed_r:T) r(r?rJrr0rr=rfind_graph_outputrArrr)rr)add_before_layernormr*rVoptional_segment_gather two_gatherrxrTrUr{need_embedding_sum_outputsum_output_indexnode_with_sum_output sum_outputis_sum_graph_outputis_sum_used_by_multiple_nodesrrrr fuse_gpt2sX         z FusionEmbedLayerNoMask.fuse_gpt2c Cs||}|dur dS|\}}|jd}|j||ddsdS||||s'dS||d|s0dS|||||d} ||| dS)aFuse embedding layer for DistilBert Args: layernorm (NodeProto): node of LayerNormalization or SkipLayerNormalization add_before_layernorm (NodeProto): the Add node before LayerNormalization, or the SkipLayerNormalization itself input_name_to_nodes (Dict[str, List[NodeProto]]): map from input name to nodes output_name_to_node (Dict[str, List[NodeProto]]): map from output name to nodes NFr$Tr)r(r?rJrqrrr) rr)rr*rVrrxrTrUrrrr fuse_distilbertds    z&FusionEmbedLayerNoMask.fuse_distilbertcCs|j|dgdg}|durdS||d}|durdS|\}}|jd} |j||dds0dS|j|dgdg} | dur@dS| d} || | |sZ||| |sTdS|} | }| } |||| scdS|| ||| |} ||| dS) aFuse embedding layer for Bert Args: layernorm (NodeProto): node of LayerNormalization or SkipLayerNormalization add_before_layernorm (NodeProto): the Add node before LayerNormalization, or the SkipLayerNormalization itself input_name_to_nodes (Dict[str, List[NodeProto]]): map from input name to nodes output_name_to_node (Dict[str, List[NodeProto]]): map from output name to nodes r7rNFr$rr#T) rr%r(r?rJrqrrr)rr)rr*rV add_2_gatherrrxryrUposition_embedding_pathrTtemprrrr fuse_berts>   z FusionEmbedLayerNoMask.fuse_bertc Cs|j|dgdg}|jdkr|durdS|d}d}nP|j|dgdg}|j|dgdg}|durG|durG|dur>dS|d}|d}n%|durh|durh|j|dgdg}|dur_dS|d}|d}n|}d}||||||rwdS|||||rdS|||||rdSdS)Nr7rrr#r$)rr%r0rrr) rrFr*rVfirst_add_pathrrr&r'rrr fuses<    zFusionEmbedLayerNoMask.fuse)r)NFN)N)__name__ __module__ __qualname____doc__r strrr rrr(rrboolrJr^r`rprqrrrrrrrrrr __classcell__rrrr rsT"  T>/H"J  b Q)2rcs8eZdZd deffdd ZddZfddZZS) FusionEmbedLayerNormalizationFrcst|d||_dS)Nz with mask)rruse_mask_index)rrrrrr rs z&FusionEmbedLayerNormalization.__init__cCs|j}t|jdkr|j|td|jn"t|jdkr1|jds1||jd<td|jn td|jdS|D]$}td|j|jdkrS|jd|jd<q<|jd kr`|jd|jd <q.z"EmbedLayerNormalization(with mask)zHEmbedLayerNormalization will not have mask since %s is not a node output) ReduceSumrPcSrrr/rrrr r4%rrr)rr@rrrrrBrCincrease_counterr?rrrr0rAnodes_to_remover)rrFr*rVrchildren_nodesrrrr rsJ                z"FusionEmbedLayerNormalization.fuse)F)rrrr rrrrrrrr rsrN)loggingrtypingrrrrr fusion_baser fusion_utilsr onnxr r r onnx_modelr rrBrrrrrr s    X