o g0@sddlmZddlmZmZddlZddlmZmZddlm Z m Z m Z m Z ddlm ZddlmZeeZGdd d ZGd d d ZdS) ) getLogger)OptionalTupleN) array_equalndarray) NodeProto TensorProtohelper numpy_helper)onnx_pb) OnnxModelc @seZdZdefddZdedeeeffddZd/defd d Z d0ded e d e ede efddZ defddZ defddZeddZed1defddZed2defddZedejfddZed3d!edefd"d#Zd$e fd%d&Zd'd(Zd)d*Zd+d,Zd-d.Zd S)4 FusionUtilsmodelcCs ||_dSN)r)selfrr\/var/www/visachat/venv/lib/python3.10/site-packages/onnxruntime/transformers/fusion_utils.py__init__s zFusionUtils.__init__ input_namereturncCsn|j|}|dur&|jjjtjkr&||\}}t d|dd|fSt d|d|dud|fS)NzCasted graph input z to int32TzDid not cast graph input z to int32: found F) rfind_graph_inputtype tensor_type elem_typerINT32cast_input_to_int32loggerdebug)rr graph_input cast_output cast_noderrrcast_graph_input_to_int32s z%FusionUtils.cast_graph_input_to_int32int32cCsf|d|}|dkrttj}n|dkrttj}n|dkr$ttj}ntd||||}||fS)N_r"float32float16z"Invalid target_type: {target_type})intrrFLOATFLOAT16 ValueError add_cast_node)rr target_type output_nameto_typer rrr cast_inputs    zFusionUtils.cast_inputNr-r, graph_namec Cs|dur |d|}|g}|dur|j}||vr,||}|r,|jdkr,|jdg}tjd||gd}|jtd|g|jj ||d|S)N _cast_to_Castr)inputsoutputsto)r/) routput_name_to_nodeop_typeinputr make_node attributeextendmake_attributeadd_node) rrr-r,r5r/r2 parent_noder rrrr*/s  zFusionUtils.add_cast_nodecCs ||dS)Nr")r.)rrrrrrJs zFusionUtils.cast_input_to_int32cCs|j}||}|D]4}|jdkr?d}|jD]}|jdkr*|jttjkr*d}nq|r?|j d}|j ||j ||q dS)Nr1Fr4Tr) rinput_name_to_nodesr6r9nameir&rroutput remove_nodereplace_input_of_all_nodes)rrr>nodesnodeis_int32attr,rrrremove_cast_int32Ms     zFusionUtils.remove_cast_int32cCs|d}|j||vr%|||j|vr%||j||t||j|}||j|<||vr7||||S|g||<|S)Nr)r7removelenappend)rEr@new_input_namer>old_input_referencerrrupdate_node_input\s   zFusionUtils.update_node_inputrc Cs<|j|}|j|}t||||}|dko|| } | S)a Before: (input)-->parent-->node-->(output) After: (input)-->parent--> | +----->node-->(output) This function returns a flag whether the parent node can be removed. r)r7r rNfind_graph_output) rrEr=r>node_input_indexparent_input_indexold_input_namerLrMparent_can_be_removedrrr skip_parentls zFusionUtils.skip_parentattribute_namecCsR|}|jD] }|j|krt|}qt|tr%t|ttfo$t||ddS||kS)aVerify that a node has expected value for an attribute. Args: node (NodeProto): a node to check attribute_name (str): name of attribute expected_value (Any): expected value of the attribute default_value (Any, optional): default value if the attribute does not exist. Defaults to None. Returns: bool: whether the check is passed or not F equal_nan)r9r?r get_attribute_value isinstancelistrr)rErUexpected_value default_valuevalueattrrrrcheck_node_attributes    z FusionUtils.check_node_attributetensorcCst|tjstdt|t|jdks|jtjjkr!td|j r@t t j |j dd|j}t |ddg}||_ |Std) zTranspose a 2-D INT8 TensorProto Args: tensor (TensorProto): tensor to be transposed Returns: tensor (TensorProto): transposed tensor z3Expected input type is an ONNX TensorProto but got z'Only INT8 2-D tensors can be transposedint8)dtyperzonly raw buffer supported)rY onnx_protorr)rrJdims data_typeINT8raw_datanumpyreshape frombuffer transposetobytes)r` int32_dataint32_transposed_datarrrtranspose_2d_int8_tensors  z$FusionUtils.transpose_2d_int8_tensorTrEcCs|jdvrtd|j||jd}|durdS|jdkp,|jdko,|jddk}|r3|s3dSt|jdkrQuantizeLinearDequantizeLinearz+Provided node is not a Q/DQ node. Op Type: rdNFrraT) r6rrget_constant_valuer7ndimshaperJrjall)rEr"allow_per_tensor_quantization_onlyscalescale_has_single_element zero_pointrrrcheck_qdq_node_for_fusions " z%FusionUtils.check_qdq_node_for_fusion input_indexcCsRt|j|ks J|j|j|}t|tr%t|ttfo$t||ddS||kS)a7Verify that a node has expected input value Args: node (NodeProto): a node to check input_index (int): index of its input to be verified expected_value (Any): expected value of the input Returns: bool: whether the check is passed or not FrV)rJr7rrtrYrZrr)rrEr}r[r]rrrcheck_node_input_values   z"FusionUtils.check_node_input_valuecCsg}|j}|jD] }|jdkr,|jd|vr,|j|jd|jd||q |rB|j|t dt |ddSdS)z>Remove Identity nodes, except those right before graph output.IdentityrzRemoved z Identity nodesN) rget_graphs_output_namesrDr6rArCr7rK remove_nodesrinforJ)rnodes_to_removegraph_output_namesrErrrremove_identity_nodess    z!FusionUtils.remove_identity_nodescC|jdSr)rremove_cascaded_cast_nodesrrrrrz&FusionUtils.remove_cascaded_cast_nodescCrr)rremove_useless_cast_nodesrrrrrrz%FusionUtils.remove_useless_cast_nodescCs@|jjdd}|dur dSg}|jD]0}|jdkrD||jd}||jd}|rD|rD||krDtd|j d|| |q|rt |j }t |j }|D]F}tt |j|@rtt |j|@st|j|jddkr|j|jd|jdnqW|j|jd|jd|j|qWdSdS) ziRemove reshape node that is not needed based on symbolic shape inference: input and output has same shapeT)updateNReshaperzRemove reshape node z* since its input shape is same as output: rd)rinfer_runtime_shaperDr6get_edge_shaper7rArrr?rKsetget_graphs_input_namesrboolrJr>replace_output_of_all_nodesrCrB)r shape_inferrrE input_shape output_shapegraph_input_namesrrrrremove_useless_reshape_nodess8  z(FusionUtils.remove_useless_reshape_nodes)r")NNN)rrr)T)__name__ __module__ __qualname__r rstrrrr!r.r&rr*rrH staticmethodrNrTr_rerrqrr|r~rrrrrrrrr sB   * r c@s(eZdZeddededefddZdS) NumpyHelperFr` fill_zerosrcCs0|rddlm}t|j|j|jdSt|S)Nr)mapping)rvrc)onnxrrrfTENSOR_TYPE_TO_NP_TYPErgr to_array)r`rrrrrr's   zNumpyHelper.to_arrayN)F)rrrrrrrrrrrrr&sr)loggingrtypingrrrjrrrrrr r r re onnx_modelr rrr rrrrrs