o g @sddlZddlZddlZddlZddlZddlmZddlmZddl m Z m Z m Z m Z mZddlZddlZddlmZmZmZmZddlZddlmZmZmZdejd ejd ejfd d Z  d0dejd ejd e eded ejf ddZGdddZ GdddZ!GdddeZ"Gdddej#dZ$GdddZ%Gddde%Z&Gddde%Z'Gd d!d!e'Z(Gd"d#d#e'Z)Gd$d%d%e'Z*Gd&d'd'ej#dZ+Gd(d)d)e+Z,dd*e"j-d+ifd,ee.efd-e e e.fd.d/Z/dS)1N)Enum)Path)DictOptionalSequenceTupleUnion) ModelProto TensorProtohelper numpy_helper) apply_plotload_model_with_shape_infersmooth_distributionpkqkreturncCs|tj|j|jd}|ddt|dd|dd|dd<|dk|dk@}d||<|dk|dk@}tj||<|S)z See https://docs.scipy.org/doc/scipy/reference/generated/scipy.special.rel_entr.html#scipy.special.rel_entr. Python implementation. dtypeNr)npemptyshaperloginf)rrresc2c1rY/var/www/visachat/venv/lib/python3.10/site-packages/onnxruntime/quantization/calibrate.pyrel_entrs2 r baseaxiscCs|dus |dks Jd|dusJdt|tj}d|tj||dd}t|tj}t||\}}d|tj||dd}t||}tj||d}|dur]|t|}||jS) z Simplifeied version of entropy. Source: https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.entropy.html. This avoids taking a dependency on scipy just for this function. Nrz0base={base} must be a positive number or `None`.z qk is None?T)r"keepdimsr") rasarrayastypefloat32sumbroadcast_arraysr rr)rrr!r"vecsrrrentropy's   r-c@sLeZdZegdZegdZddZeddZeddZ d d Z d S) TensorData)avgstdlowesthighesthist hist_edgesbins)r/r0r1r2r4cKst||_|D]E\}}|tjvr td|dtjd|tjvrJt|ds6tdt |d||j t j t j fvrJtd|j d|t|||q dS)NzUnexpected value z not in .rUnexpected type z for k=zUnexpected dtype )listkeys_attrsitemsr._allowed ValueError_floatshasattrtyperrfloat16r(setattr)selfkwargskvrrr__init__Gs   zTensorData.__init__cC4t|dr t|dstdt|d|j|jfS)Nr1r2z0Attributes 'lowest' and/or 'highest' missing in r6)r?AttributeErrordirr1r2rCrrr range_valueS zTensorData.range_valuecCrH)Nr/r0z)Attributes 'avg' and/or 'std' missing in r6)r?rIrJr/r0rKrrravg_stdYrMzTensorData.avg_stdcs$fddjD}jj|d<|S)Ncsi|]}|t|qSr)getattr).0rErKrr asz&TensorData.to_dict..CLS)r: __class____name__rCdatarrKrto_dict_s zTensorData.to_dictN) rT __module__ __qualname__ frozensetr<r>rGpropertyrLrNrWrrrrr.Cs     r.c@sjeZdZdeeeeefffddZddZ ddZ dd Z d d Z d d Z ddZddZddZdS) TensorsDatarVc Cs||_i|_|D]p\}}t|tstdt|dt|trf|tj kr;t |dkr;t |d|dd|j|<q t |dkrUt |d|d|d|dd |j|<q td |d d t |d |dt|t sutdt|d||j|<q dS)NzKeys must be strings not r6rr r1r2)r1r2r3r5zUnexpected tuple for rz , it has z elements: zValues must be TensorData not ) calibration_methodrVr; isinstancestr TypeErrorr@tupleCalibrationMethodMinMaxlenr.)rCrbrVrErFrrrrGgs"   &"  zTensorsData.__init__ccs|jEdHdSNrVrKrrr__iter__yszTensorsData.__iter__cCs ||jvSrjrkrCkeyrrr __contains__| zTensorsData.__contains__cCs |j|Srjrkrmrrr __getitem__rpzTensorsData.__getitem__cCs(||jvr td|d||j|<dS)Nz)Only an existing tensor can be modified, z is not.)rV RuntimeError)rCrnvaluerrr __setitem__s zTensorsData.__setitem__cC |jSrj)rVr9rKrrrr9rpzTensorsData.keyscCrurj)rVvaluesrKrrrrvrpzTensorsData.valuescCrurj)rVr;rKrrrr;rpzTensorsData.itemscCs|jj|j|jd}|S)N)rRrVrb)rSrTrVrbrUrrrrWs zTensorsData.to_dictN)rTrXrYrrdrr.rrGrlrorqrtr9rvr;rWrrrrr\fs r\c@seZdZdZdZdZdZdS)rgrr r]r`N)rTrXrYrhEntropy Percentile Distributionrrrrrgs rgc@sVeZdZeddZejdefddZddZ dd Z d d Z d e d e fddZ dS)CalibrationDataReadercCst|dr t|jp tS)Nget_next)r?callabler{NotImplemented)clssubclassrrr__subclasshook__sz&CalibrationDataReader.__subclasshook__rcCt)z9generate the input data dict for ONNXinferenceSession runNotImplementedErrorrKrrrr{szCalibrationDataReader.get_nextcCs|SrjrrKrrrrlzCalibrationDataReader.__iter__cCs|}|dur t|Srj)r{ StopIteration)rCresultrrr__next__szCalibrationDataReader.__next__cCrrjrrKrrr__len__rzCalibrationDataReader.__len__ start_index end_indexcCrrjr)rCrrrrr set_rangerzCalibrationDataReader.set_rangeN)rTrXrY classmethodrabcabstractmethoddictr{rlrrintrrrrrrzs rz) metaclassc@seZdZ     ddeeefdeeefddZdgfd d Z d d Z d e fddZ ddZ ddZdefddZdefddZdS)CalibraterBaseNaugmented_model.onnxF model_pathop_types_to_calibratecCsnt|tr tt||_nt|trt||_ntd||_||_||_||_ ||_ d|_ d|_ dg|_ dS)a :param model_path: ONNX model to calibrate. It should be a model file path :param op_types_to_calibrate: operator types to calibrate. By default, calibrate all the float32/float16 tensors. :param augmented_model_path: save augmented model to this path. :param symmetric: make range of tensor symmetric (central point is 0). :param use_external_data_format: use external data format to store model which size is >= 2Gb. :param per_channel: whether to compute ranges per each channel. z model_path should be model path.NCPUExecutionProvider)rcrdrrmodelr=raugmented_model_path symmetricuse_external_data_format per_channel augment_model infer_sessionexecution_providers)rCrrrrrrrrrrGs    zCalibraterBase.__init__rcCs||_|dS)zz reset the execution providers to execute the collect_data. It triggers to re-creating inference session. N)rcreate_inference_session)rCrrrrset_execution_providerss z&CalibraterBase.set_execution_providerscCs,t}tjj|_tj|j||jd|_dS)z9 create an OnnxRuntime InferenceSession. ) sess_options providersN) onnxruntimeSessionOptionsGraphOptimizationLevelORT_DISABLE_ALLgraph_optimization_levelInferenceSessionrrr)rCrrrrrs  z'CalibraterBase.create_inference_sessionrc Csdd|jjD}|dd|jjD|dd|jjDdd|jjD}t}tjtj h}|jj D]4}|j rB|j |j vrkt |j|jD] }||vrj||}|jdrj|jjj|vrj||vrj||qJq7||fS)z select input/output tensors of candidate nodes to calibrate. returns: tensors (set): set of tensor name. value_infos (dict): tensor name to value info. cSi|]}|j|qSrnamerPvirrrrQz>CalibraterBase.select_tensors_to_calibrate..cSrrr)rPotrrrrQrcSrrr)rPitrrrrQrcSh|]}|jqSrr)rPinitrrr z=CalibraterBase.select_tensors_to_calibrate.. tensor_type)graph value_infoupdateoutputinput initializersetr FLOATFLOAT16noderop_type itertoolschainr@HasFieldr elem_typeadd) rCr value_infosrtensors_to_calibratetensor_type_to_calibrater tensor_namerrrrselect_tensors_to_calibrates$    z*CalibraterBase.select_tensors_to_calibratecC|jS)zP return: augmented onnx model. Call after calling augment_graph )rrKrrrget_augment_modelsz CalibraterBase.get_augment_modelcCr)z abstract method: augment the input model to prepare for collecting data. It will: 1. augment the model to be able to collect desired statistics data 2. save augmented model to augmented_model_paths rrKrrr augment_graphszCalibraterBase.augment_graph data_readercCr)z abstract method: collect the tensors that will be used for range computation. It can be called multiple times. r)rCrrrr collect_datazCalibraterBase.collect_datarcCr)ze abstract method: compute data based on the calibration method stored in TensorsData rrKrrr compute_data"rzCalibraterBase.compute_data)NrFFF)rTrXrYrrdrrrrGrrr rrrrzrr\rrrrrrs$   " rcs|eZdZ        ddeeefdeeeffdd Zd d Z d d Z d e fddZ ddZ defddZZS)MinMaxCalibraterNrF{Gz?rrc s|tj|||||| dg|_d|_t|jjj|_dd|jjjD|_ ||_ |r6|dks2|dkr6t d||_ ||_ dS)aw :param model_path: ONNX model to calibrate. It is a model path :param op_types_to_calibrate: operator types to calibrate. By default, calibrate all the float32/float16 tensors. :param augmented_model_path: save augmented model to this path. :param symmetric: make range of tensor symmetric (central point is 0). :param use_external_data_format: use external data format to store model which size is >= 2Gb :param moving_average: compute the moving average of the minimum and maximum values instead of the global minimum and maximum. :param averaging_constant: constant smoothing factor to use when computing the moving average. :param max_intermediate_outputs: maximum number of intermediate outputs before an intermediate range is computed. :param per_channel: whether to compute ranges per each channel. )rrrrrNcSrrrrPrrrrrLrz,MinMaxCalibrater.__init__..rr z;Invalid averaging constant, which should not be < 0 or > 1.)superrGintermediate_outputscalibrate_tensors_rangerirrrnum_model_outputsmodel_original_outputsmoving_averager=averaging_constantmax_intermediate_outputs) rCrrrrrrrrrrSrrrG*s" zMinMaxCalibrater.__init__csj\}}ttttjdgtj d}jj j |ddfdd}|D] }||d||dq1t jjjjd d S) z Adds ReduceMin and ReduceMax nodes to all quantization_candidates op type nodes in model and ensures their outputs are stored as part of the graph output :return: augmented ONNX model rcSs6|jD]}tj||jr|jSqtd|d)Nz&Model does not contain a version for 'z'.) opset_importonnxdefshasdomainversionrr)rrrrrrget_op_version^s  z6MinMaxCalibrater.augment_graph..get_op_versionc sd}|d|}|d}tjj||g|g||d}tjjd|g|g|d}ddjjjD}|d djjjD|d djjjD||vrV||j j j }nt d |d j rt||j j jj} d gtd| } |jdkr|jtd| n tt} ttj| tjd| } |j| jjj| jjj||gjjjt ||dgdS)Nr __Reshape)r$rReshape)inputsoutputsrcSrrrrrrrrQwrzNMinMaxCalibrater.augment_graph..add_reduce_min_max..cSrrr)rPorrrrQxrcSrrrrPirrrrQyrz'Unable to guess tensor type for tensor zE, running shape inference before quantization may resolve this issue.rr]axesr)!rr make_noderrrrrrr@rrr=rrirdimrange attributeappendmake_attributerduuiduuid4r from_arrayrarrayint64rrextendmake_tensor_value_info) rreduce_op_namer$ reduce_outputintermediate_output reduce_node reshape_noder onnx_type tensor_rank reduced_axesreduce_axes_name reduce_axesrreshape_shape_namerCrradd_reduce_min_maxds>     z:MinMaxCalibrater.augment_graph..add_reduce_min_max ReduceMin ReduceMaxsave_as_external_dataN)rrrdrrr rrrrrrrrsaverr)rCtensorsr reshape_shaper tensorrrrrSs .   zMinMaxCalibrater.augment_graphcC g|_dSrjrrKrrrclear_collected_datarpz%MinMaxCalibrater.clear_collected_datarcCs |}|sn|j|jd||jdur$t|j|jkr$|qt|jdkr5|jdur5t d| }t |t sHt dt|d|dS)NTrNo data is collected.z+compute_data must return a TensorsData not r6)r{rrrrunrrirrr=rrcr\rer@)rCrrtrrrrs    zMinMaxCalibrater.collect_datac Cs|s|S|D]o\}}t|tr|jd}|jd}n|\}}t||tr6||jd}||jd}n||\}}|jrR||j||} ||j||} n t||} t||} t|tsht||trqt| | d||<q| | f||<q|S)Nrr r^)r;rcr.rLrrminmax) rC old_range new_rangernrsold_minold_maxnew_minnew_max min_value max_valuerrr merge_ranges(      zMinMaxCalibrater.merge_rangerc stjdkr jSfddttjdDfddjD}i|D]}|D] \}}|g|q.q(jdfddtdtdD}fdd D}g}tdtdD]\}jrt j ||dd } t j ||d dd } nt j ||dd } t j ||d dd } j rt j t | t | gdd } |t| | gqg|t| | gqgttjtt||} jr݈j| _jS| _jS) z Compute the min-max range of tensor :return: dictionary mapping: {added node names: (ReduceMin, ReduceMax) pairs } rcsg|] }j|jqSr)r get_outputsrrrKrr sz1MinMaxCalibrater.compute_data..cg|] }tt|qSrrziprPr output_namesrrr$Ncsg|] }|ddqS)rr) rpartitionr)added_output_namesrrr$sr]cs i|] }|jvr||qSr)rr)merged_output_dictrCrrrQsz1MinMaxCalibrater.compute_data..r%r )rirrrr; setdefaultrrrrmeanrrrabsrfr\rgrhrr'r") rCoutput_dicts_listdrErFcalibrate_tensor_namesmerged_added_output_dictpairsrmin_value_arraymax_value_arraymax_absolute_valuenew_calibrate_tensors_ranger)r-r.r*rCrrsF    zMinMaxCalibrater.compute_data)NrFFFrNF)rTrXrYrrdrrrrGrrrzrr"r\r __classcell__rrrrr)s&  )I!rcsveZdZ         dd eeefd eeeffd d Zd dZ ddZ de fddZ de fddZZS)HistogramCalibraterNrF percentile-X@samerrc svtj|||||dg|_d|_t|jjj|_dd|jjjD|_ d|_ ||_ ||_ ||_ | |_d|_| |_dS)a= :param model_path: ONNX model to calibrate. It is a model path. :param op_types_to_calibrate: operator types to calibrate. By default, calibrate all the float32/float16 tensors. :param augmented_model_path: save augmented model to this path. :param use_external_data_format: use external data format to store model which size is >= 2Gb :param method: A string. One of ['entropy', 'percentile']. :param symmetric: make range of tensor symmetric (central point is 0). :param num_bins: number of bins to create a new histogram for collecting tensor values. :param num_quantized_bins: number of quantized bins. Default 128. :param percentile: A float number between [0, 100]. Default 99.99. :param scenario: see :class:`DistributionCalibrater` )rrrrNcSrrrrrrrr+rz/HistogramCalibrater.__init__..)rrGrrrirrrrr collectormethodnum_binsnum_quantized_binsr=rscenario) rCrrrrrCrrDrEr=rFrrrrGs$ zHistogramCalibrater.__init__cCsV||j\|_}|jD]}||jvr|jjj||q tj|j|j |j ddS)z make all quantization_candidates op type nodes as part of the graph output. :return: augmented ONNX model r N) rrrrrrrrrrr)rCrrrrrr4s   z!HistogramCalibrater.augment_graphcCrrjrrKrrrrDrpz(HistogramCalibrater.clear_collected_datarc sBddjD}ddjD |}|sn-jd|}g}t|D]\}}||vr<|t|q)||q)j|qt jdkrTt dfd djD}i|D]} | D] \} } | g| qhqbfd d D} j stjjjjjjd _ j | dS) zy Entropy Calibrator collects operators' tensors as well as generates tensor histogram for each operator. cSrrrrPnode_argrrrrKrz3HistogramCalibrater.collect_data..cSsg|]}|jqSrrrGrrrr$Lrz4HistogramCalibrater.collect_data..TNrrcr%rr&r(r)rrr$br+cs i|] }|jvr||qSr)rr) merged_dictrCrrrQks z4HistogramCalibrater.collect_data..)rCrrDrEr=rF)r get_inputsr#r{r enumeratercopyrrir=r;r/rBHistogramCollectorrCrrDrEr=rFcollectr) rCrinput_names_setrr fixed_outputs output_indexrr2r3rErFclean_merged_dictr)rIr*rCrrGsH      z HistogramCalibrater.collect_datarcCsh|jstdt|trtj}nt|trtj}nt|tr"tj }n t dt |dt ||j S)z Compute the min-max range of tensor :return: dictionary mapping: {tensor name: (min value, max value)} z9No collector created and can't generate calibration data.zUnknown calibrater z". This method must be overwritten.)rBr=rcEntropyCalibraterrgrwPercentileCalibraterrxDistributionCalibraterryrer@r\compute_collection_result)rCcalrrrrzs   z HistogramCalibrater.compute_data) NrFr=Fr>r?r@rA)rTrXrYrrdrrrrGrrrzrr\rr;rrrrr<s&  ,3r<csFeZdZ       d deeefdeeeffdd ZZ S) rSNrFr-r>rrc  tj||||||||ddS)a :param model_path: ONNX model to calibrate. It is a model path :param op_types_to_calibrate: operator types to calibrate. By default, calibrate all the float32/float16 tensors. :param augmented_model_path: save augmented model to this path. :param use_external_data_format: use external data format to store model which size is >= 2Gb :param method: A string. One of ['entropy', 'percentile', 'distribution']. :param symmetric: make range of tensor symmetric (central point is 0). :param num_bins: number of bins to create a new histogram for collecting tensor values. :param num_quantized_bins: number of quantized bins. Default 128. )rCrrDrENrrG) rCrrrrrCrrDrErrrrG zEntropyCalibrater.__init__)NrFr-Fr>r> rTrXrYrrdrrrrGr;rrrrrS  rScsFeZdZ       d deeefdeeeffd d ZZ S) rTNrFr=r?r@rrc rX)a :param model_path: ONNX model to calibrate. It is a model path :param op_types_to_calibrate: operator types to calibrate. By default, calibrate all the float32/float16 tensors. :param augmented_model_path: save augmented model to this path. :param use_external_data_format: use external data format to store model which size is >= 2Gb :param method: A string. One of ['entropy', 'percentile', 'distribution']. :param symmetric: make range of tensor symmetric (central point is 0). :param num_quantized_bins: number of quantized bins. Default 128. :param percentile: A float number between [0, 100]. Default 99.99. )rCrrDr=NrY) rCrrrrrCrrDr=rrrrGrZzPercentileCalibrater.__init__)NrFr=Fr?r@r[rrrrrTr\rTcsDeZdZ      d deeefdeeeffd d ZZ S) rUNrF distributionr>rArrc stj|||||||ddS)a :param model_path: ONNX model to calibrate. It is a model path :param op_types_to_calibrate: operator types to calibrate. By default, calibrate all the float32/float16 tensors. :param augmented_model_path: save augmented model to this path. :param use_external_data_format: use external data format to store model which size is >= 2Gb :param method: A string. One of ['entropy', 'percentile', 'distribution']. :param symmetric: make range of tensor symmetric (central point is 0). :param num_bins: number of bins to create a new histogram for collecting tensor values. :param scenario: for float 8 only, if `scenario="same"`, the algorithm weights and float 8 follow the same distribution, if `scenario="p3"`, it assumes the weights follow a gaussian law and float 8 ~ X^3 where X is a gaussian law )rCrDrFNrY)rCrrrrrCrDrFrrrrGs zDistributionCalibrater.__init__)NrFr]r>rAr[rrrrrUs  rUc@s,eZdZdZejddZejddZdS)CalibrationDataCollectorzL Base class for collecting data for calibration-based quantization. cCr)z Generate informative data based on given data. name_to_arr : dict tensor name to NDArray data rrC name_to_arrrrrrNsz CalibrationDataCollector.collectcCr)z? Get the optimal result among collection data. rrKrrrrVsz2CalibrationDataCollector.compute_collection_resultN)rTrXrY__doc__rrrNrVrrrrr^s  r^c@sveZdZdZddZddZddZdd Zd d Zd d Z ddZ ddZ ddZ e dddZddZddZdS)rMa` Collecting histogram for each tensor. Percentile and Entropy method are supported. ref: https://github.com//apache/incubator-mxnet/blob/master/python/mxnet/contrib/quantization.py ref: https://docs.nvidia.com/deeplearning/tensorrt/pytorch-quantization-toolkit/docs/_modules/ pytorch_quantization/calib/histogram.html cCs.i|_||_||_||_||_||_||_dSrj)histogram_dictrCrrDrEr=rF)rCrCrrDrEr=rFrrrrGs zHistogramCollector.__init__cCrrj)rbrKrrrget_histogram_dictsz%HistogramCollector.get_histogram_dictcCsHtd|jdvr||S|jdkr |jr||S||Std)Nz/Collecting tensor data and making histogram ...>r-r]r=DOnly 'entropy', 'percentile' or 'distribution' methods are supported)printrC collect_valuercollect_absolute_valuer=r_rrrrNs     zHistogramCollector.collectcCsv|D]3\}}t|trD|D]}t|tjs$Jdt|d|qtdd|D}t|dks>Jd|d|t|}nt|tjsVt dt|d||}| }|j dkrlt |}t |}ntjd|jd}tjd|jd}t|}||jvrtj||jd \} } | |j} |jtjksJd | | ||f|j|<q|j|} | d } | d } t| d sJdt| t| d sJdt| | d}| d}t |}||dkr|d|d}t|d||||}t||f}tj||d \} } | |j} | dt||7<|jtjks)Jd | | t | |t | |f|j|<qdS)z5 Collect histogram on absolute value r7z for tensor=css|]}|jVqdSrjr)rParrr 6sz.r z6The calibration expects only one element type but got rr)r5zMonly float32 or float16 is supported, every constant must be explicitly typedr]r`rz'old_min should be a numpy array but is rN)r;rcr8rndarrayr@rrir&r=flattensizerrrrabsoluterb histogramrDr'float64r?arangehstack)rCr`rdata_arrarrdtypes data_arr_npr r!r3r4 old_histogramrrold_histold_hist_edges temp_amaxwidth new_bin_edgesrrrrg.s\ &            z)HistogramCollector.collect_absolute_valuec Cs|D]k\}}t|}|}|jdkr!t|}t|}ntjd|jd}tjd|jd}tjtt |t ||jd}||j vrW|j |}| ||||||j |<qtj ||j | |fd\}} || |||f|j |<qdS)z1 Collect histogram on real value rrrN)r;rr&rkrlrrrrr1rbmerge_histogramrnrD) rCr`rrrr r! thresholdrvr3r4rrrrfhs,         z HistogramCollector.collect_valuecCs|\}}}} } || kr)tj|t|| | fd\} } | ||t||t| || fS| dkrAtj|t|| |fd\} }| |7} n9t|}d| |}t|| |d}|d|}||| }tj||| |fd\} }| ||||7<| |t||t| ||fS)Nr|rr]r )rrnrirrr)rCrvrrrr new_thresholdrwrxrr old_thresholdnew_histrr3r4 old_num_bins old_stridehalf_increased_bins new_num_binsrrrr}s2    z"HistogramCollector.merge_histogramcCsl|jr t|jdkrtdtd|jd|jdkr |S|jdkr)|S|jdkr2|Std) Nrz=Histogram has not been collected. Please run collect() first.z0Finding optimal threshold for each tensor using z algorithm ...r-r=r]rd)rbrir=rerCcompute_entropycompute_percentilecompute_distributionrKrrrrVs   z,HistogramCollector.compute_collection_resultcCs|jdks |jdkrtd|j}|j}i}tdt|td|jtdd|d|d |D]\}}|d}|d }|}t ||} |j rrt | |d} tj || |j d  tj || |j d f||<n*d|d } t | d | } t | | } tj || |j d tj || |j d f||<|d} |d}||d| kr| ||d f||<||d |kr||d|f||<g|||ddR||<tjdddvrt||q8|S)Nrdz     z'HistogramCollector.compute_distributioncs|d}|d}|j}|d}|d}|djt||d}fddt|jD} t||ddD]} || } t|| d|} || || f| | |<t|| | } | }t|d| }t|| d}|d|7<|d|7<|dk tj }tj|tj d}| j|}t|D]}||}||}t| ||||<q|dt| ||d7<tj|jtj d}t|D] }||}||}t|||}|dkr||||||<qt |}t |}|dus|durtj tj d}n tj t||d}||| |<q5t|}| |}|d}|d }|d|kr2||df}|d|kr?|d|f}t|dd sIJt|dd sSJ|S) aFGiven a dataset, find the optimal threshold for quantizing it. The reference distribution is `q`, and the candidate distribution is `p`. `q` is a truncated version of the original distribution. Ref: http://on-demand.gputechconf.com/gtc/2017/presentation/s7310-8-bit-inference-with-tensorrt.pdf rr r]cs(g|]}tjddtjddfqS)rr)rrrrrrr$Es(z.Nrrr`r)rlrrzerosrrrLdeepcopyr)r'rrrrr-argminr?)rCrnrEr3r4rDzero_bin_indexnum_half_quantized_bin kl_divergence thresholdsrrrsliced_distributionpleft_outliers_countright_outliers_countnonzerosquantized_binsnum_merged_binsindexstartendqnormdivmin_kl_divergence_idxrr r!rrrr7sf        z(HistogramCollector.get_entropy_thresholdN)r )rTrXrYrarGrcrNrgrfr}rVrr staticmethodrrrrrrrrM s :  .  (rMrFrrc CsNd}|tjkr3|dd}|dd}|dd} |dd} |dd} t||||||| | | d }na|tjkrV|d d } |d d } |dd}t|||||| | d }n>|tjkry|d d } |dd}|dd}t|||||| |d}n|tjkr|d d } |dd}t ||||| |d}|r| | |St d|)NrFrrrrr)rrrrrrrDr>rE)rrrDrEr?r=r@T)rrrDr=rFrA)rrDrFzUnsupported calibration method ) rgrhrrrwrSrxrTryrUrrr=)rrrcalibrate_methodr extra_options calibratorrrrrrrDrEr=rFrrrcreate_calibratorsx               r)Nr)0rrLrrrenumrpathlibrtypingrrrrrnumpyrrr r r r r quant_utilsrrrrjr floatrr-r.r\rgABCMetarzrrr<rSrTrUr^rMrhrdrrrrrsl   #4n_""#