o JjgG@sdZddlmZddlZddlmZmZddlmZddl m Z m Z m Z m Z mZddlmZddlmZdd lmZdd lmZdd lmZeeZGd d d eeZGdddeZGdddZGdddeeZ GdddeeZ!GdddeeZ"dS)z3Interfaces to be implemented by general evaluators.) annotationsN)ABCabstractmethod)Enum)AnyOptionalSequenceTupleUnion)warn) AgentAction)BaseLanguageModel)run_in_executor)Chainc@seZdZdZdZ dZ dZ dZ dZ dZ dZ d Z d Z d Z d Z d Z dZ dZ dZ dZ dZ dZ dZ dZdS) EvaluatorTypezThe types of the evaluators.qacot_qa context_qapairwise_string score_stringlabeled_pairwise_stringlabeled_score_string trajectorycriterialabeled_criteriastring_distance exact_match regex_matchpairwise_string_distanceembedding_distancepairwise_embedding_distance json_validity json_equalityjson_edit_distancejson_schema_validationN)__name__ __module__ __qualname____doc__QACOT_QA CONTEXT_QAPAIRWISE_STRING SCORE_STRINGLABELED_PAIRWISE_STRINGLABELED_SCORE_STRINGAGENT_TRAJECTORYCRITERIALABELED_CRITERIASTRING_DISTANCE EXACT_MATCH REGEX_MATCHPAIRWISE_STRING_DISTANCEEMBEDDING_DISTANCEPAIRWISE_EMBEDDING_DISTANCE JSON_VALIDITY JSON_EQUALITYJSON_EDIT_DISTANCEJSON_SCHEMA_VALIDATIONr=r=S/var/www/html/zoom/venv/lib/python3.10/site-packages/langchain/evaluation/schema.pyrsTrc@s"eZdZdZeed ddZd S) LLMEvalChainz,A base class for evaluators that use an LLM.llmr kwargsrreturncKdS)z#Create a new evaluator from an LLM.Nr=)clsr@rAr=r=r>from_llmMzLLMEvalChain.from_llmN)r@r rArrBr?)r%r&r'r( classmethodrrEr=r=r=r>r?Js r?c@sXeZdZdZedddZedddZedd d Zedd d Z ddddZ d S)_EvalArgsMixinz(Mixin for checking evaluation arguments.rBboolcCrCz2Whether this evaluator requires a reference label.Fr=selfr=r=r>requires_referenceVz!_EvalArgsMixin.requires_referencecCrC)0Whether this evaluator requires an input string.Fr=rKr=r=r>requires_input[rNz_EvalArgsMixin.requires_inputstrcCd|jjdS)z&Warning to show when input is ignored.zIgnoring input in , as it is not expected. __class__r%rKr=r=r>_skip_input_warning`sz"_EvalArgsMixin._skip_input_warningcCrR)z*Warning to show when reference is ignored.zIgnoring reference in rSrTrKr=r=r>_skip_reference_warningesz&_EvalArgsMixin._skip_reference_warningN reference Optional[str]inputNonecCs||jr|durt|jjd|dur|jst|j|jr,|dur,t|jjd|dur:|js_check_evaluation_argsls z%_EvalArgsMixin._check_evaluation_argsrBrIrBrQ)NN)rXrYrZrYrBr[) r%r&r'r(propertyrMrPrVrWr]r=r=r=r>rHSs    rHc@sxeZdZdZedddZedddZed d d dddZd d d dddZ d d d dddZ d d d dddZ d S) StringEvaluatorzcGrade, tag, or otherwise evaluate predictions relative to their inputs and/or reference labels.rBrQcCs|jjS)zThe name of the evaluation.rTrKr=r=r>evaluation_nameszStringEvaluator.evaluation_namerIcCrCrJr=rKr=r=r>rMrNz"StringEvaluator.requires_referenceNrXrZ predictionUnion[str, Any]rXOptional[Union[str, Any]]rZrArdictcKrC)a:Evaluate Chain or LLM output, based on optional input and label. Args: prediction (str): The LLM or chain prediction to evaluate. reference (Optional[str], optional): The reference label to evaluate against. input (Optional[str], optional): The input to consider during evaluation. kwargs: Additional keyword arguments, including callbacks, tags, etc. Returns: dict: The evaluation results containing the score or value. It is recommended that the dictionary contain the following keys: - score: the score of the evaluation, if applicable. - value: the string value of the evaluation, if applicable. - reasoning: the reasoning for the evaluation, if applicable. Nr=rLrdrXrZrAr=r=r>_evaluate_stringsrFz!StringEvaluator._evaluate_stringscs$td|jf|||d|IdHS)aIAsynchronously evaluate Chain or LLM output, based on optional input and label. Args: prediction (str): The LLM or chain prediction to evaluate. reference (Optional[str], optional): The reference label to evaluate against. input (Optional[str], optional): The input to consider during evaluation. kwargs: Additional keyword arguments, including callbacks, tags, etc. Returns: dict: The evaluation results containing the score or value. It is recommended that the dictionary contain the following keys: - score: the score of the evaluation, if applicable. - value: the string value of the evaluation, if applicable. - reasoning: the reasoning for the evaluation, if applicable. NrdrXrZ)rrirhr=r=r>_aevaluate_stringss z"StringEvaluator._aevaluate_stringsrYcKs&|j||d|jd|||d|S)aEvaluate Chain or LLM output, based on optional input and label. Args: prediction (str): The LLM or chain prediction to evaluate. reference (Optional[str], optional): The reference label to evaluate against. input (Optional[str], optional): The input to consider during evaluation. kwargs: Additional keyword arguments, including callbacks, tags, etc. Returns: dict: The evaluation results containing the score or value. rcrjNr=)r]rirhr=r=r>evaluate_stringss z StringEvaluator.evaluate_stringscs.|j||d|jd|||d|IdHS)a Asynchronously evaluate Chain or LLM output, based on optional input and label. Args: prediction (str): The LLM or chain prediction to evaluate. reference (Optional[str], optional): The reference label to evaluate against. input (Optional[str], optional): The input to consider during evaluation. kwargs: Additional keyword arguments, including callbacks, tags, etc. Returns: dict: The evaluation results containing the score or value. rcrjNr=)r]rkrhr=r=r>aevaluate_stringss z!StringEvaluator.aevaluate_stringsr_r^) rdrerXrfrZrfrArrBrg) rdrQrXrYrZrYrArrBrg) r%r&r'r(r`rbrMrrirkrlrmr=r=r=r>ras&  #rac@s\eZdZdZeddddddZddddddZddddddZddddddZdS)PairwiseStringEvaluatorzDCompare the output of two models (or two outputs of the same model).NrcrdrQ prediction_brXrYrZrArrBrgcKrC)1Evaluate the output string pairs. Args: prediction (str): The output string from the first model. prediction_b (str): The output string from the second model. reference (Optional[str], optional): The expected output / reference string. input (Optional[str], optional): The input string. kwargs: Additional keyword arguments, such as callbacks and optional reference strings. Returns: dict: A dictionary containing the preference, scores, and/or other information. Nr=rLrdrorXrZrAr=r=r>_evaluate_string_pairsrFz.PairwiseStringEvaluator._evaluate_string_pairscs&td|jf||||d|IdHS)@Asynchronously evaluate the output string pairs. Args: prediction (str): The output string from the first model. prediction_b (str): The output string from the second model. reference (Optional[str], optional): The expected output / reference string. input (Optional[str], optional): The input string. kwargs: Additional keyword arguments, such as callbacks and optional reference strings. Returns: dict: A dictionary containing the preference, scores, and/or other information. NrdrorXrZ)rrrrqr=r=r>_aevaluate_string_pairss z/PairwiseStringEvaluator._aevaluate_string_pairscKs(|j||d|jd||||d|S)rprcrtNr=)r]rrrqr=r=r>evaluate_string_pairs/sz-PairwiseStringEvaluator.evaluate_string_pairscs0|j||d|jd||||d|IdHS)rsrcrtNr=)r]rurqr=r=r>aevaluate_string_pairsLs z.PairwiseStringEvaluator.aevaluate_string_pairs) rdrQrorQrXrYrZrYrArrBrg) r%r&r'r(rrrrurvrwr=r=r=r>rns#"rnc@sbeZdZdZedddZedddddZdddddZdddddZ dddddZ dS)AgentTrajectoryEvaluatorz,Interface for evaluating agent trajectories.rBrIcCrC)rOTr=rKr=r=r>rPmrNz'AgentTrajectoryEvaluator.requires_inputN)rXrdrQagent_trajectory!Sequence[Tuple[AgentAction, str]]rZrXrYrArrgcKrC)Evaluate a trajectory. Args: prediction (str): The final predicted response. agent_trajectory (List[Tuple[AgentAction, str]]): The intermediate steps forming the agent trajectory. input (str): The input to the agent. reference (Optional[str]): The reference answer. Returns: dict: The evaluation result. Nr=rLrdryrZrXrAr=r=r>_evaluate_agent_trajectoryrrFz3AgentTrajectoryEvaluator._evaluate_agent_trajectorycs&td|jf||||d|IdHS)Asynchronously evaluate a trajectory. Args: prediction (str): The final predicted response. agent_trajectory (List[Tuple[AgentAction, str]]): The intermediate steps forming the agent trajectory. input (str): The input to the agent. reference (Optional[str]): The reference answer. Returns: dict: The evaluation result. N)rdryrXrZ)rr}r|r=r=r>_aevaluate_agent_trajectorys z4AgentTrajectoryEvaluator._aevaluate_agent_trajectorycKs(|j||d|jd||||d|S)r{rcrdrZryrXNr=)r]r}r|r=r=r>evaluate_agent_trajectorysz2AgentTrajectoryEvaluator.evaluate_agent_trajectorycs0|j||d|jd||||d|IdHS)r~rcrNr=)r]rr|r=r=r>aevaluate_agent_trajectorys z3AgentTrajectoryEvaluator.aevaluate_agent_trajectoryr^) rdrQryrzrZrQrXrYrArrBrg) r%r&r'r(r`rPrr}rrrr=r=r=r>rxjs %$rx)#r( __future__rloggingabcrrenumrtypingrrrr r warningsr langchain_core.agentsr langchain_core.language_modelsr langchain_core.runnables.configrlangchain.chains.baser getLoggerr%loggerrQrr?rHrarnrxr=r=r=r>s$        6 1tr