o Jjg4@sTdZddlmZmZmZmZmZmZmZddl m Z ddl m Z ddl mZddlmZddlmZmZddlmZmZdd lmZmZmZdd lmZdd lmZdd l m!Z!m"Z"dd l#m$Z%eeeegeeee&ffZ'eeeeeegeeee&ffZ(GdddeZ)Gddde)Z*ee'ee"fZ+ee!e,e)fZ-GdddeZ.dS)z!Configuration for run evaluators.)AnyCallableDictListOptionalSequenceUnion) Embeddings)BaseLanguageModel)BasePromptTemplate) RunEvaluator)EvaluationResultEvaluationResults)ExampleRun) BaseModel ConfigDictField) CRITERIA_TYPE)EmbeddingDistance) EvaluatorTypeStringEvaluator)StringDistancec@s0eZdZUdZeed<deeeffddZ dS) EvalConfigaConfiguration for a given run evaluator. Parameters ---------- evaluator_type : EvaluatorType The type of evaluator to use. Methods ------- get_kwargs() Get the keyword arguments for the evaluator configuration. evaluator_typereturncCs2i}|D]\}}|dkr q|durq|||<q|S)zGet the keyword arguments for the load_evaluator call. Returns ------- Dict[str, Any] The keyword arguments for the load_evaluator call. rN)selfkwargsfieldvalrrY/var/www/html/zoom/venv/lib/python3.10/site-packages/langchain/smith/evaluation/config.py get_kwargs0s  zEvalConfig.get_kwargsN) __name__ __module__ __qualname____doc__r__annotations__rstrrr"rrrr!rs rcsfeZdZUdZdZeeed< dZeeed< dZ eeed< de ee fffdd Z Z S) SingleKeyEvalConfigzBConfiguration for a run evaluator that only requires a single key.N reference_keyprediction_key input_keyrcs$t}dD]}||dq|S)N)r*r+r,)superr"pop)rrkey __class__rr!r"Qs zSingleKeyEvalConfig.get_kwargs)r#r$r%r&r*rr(r'r+r,rrr" __classcell__rrr0r!r)Cs "r)c@seZdZUdZeedZeee e fe d< dZ e ee e d< dZe eee d< dZe ee d< dZe ee d< dZe ee d < dZe ee d < ed d ZGd ddeZGdddeZGdddeZGdddeZGdddeZGdddeZGdddeZGdddeZ Gddde!Z"Gdd d eZ#Gd!d"d"eZ$Gd#d$d$eZ%Gd%d&d&e%Z&dS)' RunEvalConfiga0Configuration for a run evaluation. Parameters ---------- evaluators : List[Union[EvaluatorType, EvalConfig, RunEvaluator, Callable]] Configurations for which evaluators to apply to the dataset run. Each can be the string of an :class:`EvaluatorType `, such as EvaluatorType.QA, the evaluator type string ("qa"), or a configuration for a given evaluator (e.g., :class:`RunEvalConfig.QA `). custom_evaluators : Optional[List[Union[RunEvaluator, StringEvaluator]]] Custom evaluators to apply to the dataset run. reference_key : Optional[str] The key in the dataset run to use as the reference string. If not provided, it will be inferred automatically. prediction_key : Optional[str] The key from the traced run's outputs dictionary to use to represent the prediction. If not provided, it will be inferred automatically. input_key : Optional[str] The key from the traced run's inputs dictionary to use to represent the input. If not provided, it will be inferred automatically. eval_llm : Optional[BaseLanguageModel] The language model to pass to any evaluators that use a language model. )default_factory evaluatorsNcustom_evaluatorsbatch_evaluatorsr*r+r,eval_llmTarbitrary_types_allowedcfeZdZUdZdZeeed<dZee ed<e j Z e ed< d deede ddffdd ZZS) zRunEvalConfig.Criteriaa"Configuration for a reference-free criteria evaluator. Parameters ---------- criteria : Optional[CRITERIA_TYPE] The criteria to evaluate. llm : Optional[BaseLanguageModel] The language model to use for the evaluation chain. Ncriteriallmrrrc tjdd|i|dSNr<rr-__init__rr<rr0rr!rAzRunEvalConfig.Criteria.__init__N)r#r$r%r&r<rrr'r=r rCRITERIArrrAr2rrr0r!Criterias  rFcr;) zRunEvalConfig.LabeledCriteriaa,Configuration for a labeled (with references) criteria evaluator. Parameters ---------- criteria : Optional[CRITERIA_TYPE] The criteria to evaluate. llm : Optional[BaseLanguageModel] The language model to use for the evaluation chain. Nr<r=rrrc r>r?r@rBr0rr!rArCz&RunEvalConfig.LabeledCriteria.__init__rD)r#r$r%r&r<rrr'r=r rLABELED_CRITERIArrrAr2rrr0r!LabeledCriterias  rHc@sJeZdZUdZejZeed<dZe e ed<dZ e e ed<e ddZdS)zRunEvalConfig.EmbeddingDistanceaIConfiguration for an embedding distance evaluator. Parameters ---------- embeddings : Optional[Embeddings] The embeddings to use for computing the distance. distance_metric : Optional[EmbeddingDistanceEnum] The distance metric to use for computing the distance. rN embeddingsdistance_metricTr9)r#r$r%r&rEMBEDDING_DISTANCErr'rIrr rJEmbeddingDistanceEnumr model_configrrrr!rs   rc@s>eZdZUdZejZeed<dZe e ed< dZ e ed<dS)zRunEvalConfig.StringDistancezConfiguration for a string distance evaluator. Parameters ---------- distance : Optional[StringDistanceEnum] The string distance metric to use. rNdistanceTnormalize_score) r#r$r%r&rSTRING_DISTANCErr'rNrStringDistanceEnumrOboolrrrr!rs   rc@@eZdZUdZejZeed<dZe e ed<dZ e e ed<dS)zRunEvalConfig.QAa-Configuration for a QA evaluator. Parameters ---------- prompt : Optional[BasePromptTemplate] The prompt template to use for generating the question. llm : Optional[BaseLanguageModel] The language model to use for the evaluation chain. rNr=prompt) r#r$r%r&rQArr'r=rr rTr rrrr!rUs  rUc@rS)zRunEvalConfig.ContextQA<Configuration for a context-based QA evaluator. Parameters ---------- prompt : Optional[BasePromptTemplate] The prompt template to use for generating the question. llm : Optional[BaseLanguageModel] The language model to use for the evaluation chain. rNr=rT r#r$r%r&r CONTEXT_QArr'r=rr rTr rrrr! ContextQA  rYc@rS)zRunEvalConfig.CoTQArVrNr=rTrWrrrr!CoTQArZr[c@ eZdZUdZejZeed<dS)zRunEvalConfig.JsonValidityz\Configuration for a json validity evaluator. Parameters ---------- rN)r#r$r%r&r JSON_VALIDITYrr'rrrr! JsonValidity& r^c@r\)z#RunEvalConfig.JsonEqualityEvaluatorz\Configuration for a json equality evaluator. Parameters ---------- rN)r#r$r%r&r JSON_EQUALITYrr'rrrr!JsonEqualityEvaluator/r_rac@sDeZdZUdZejZeed<dZe ed<dZ e ed<dZ e ed<dS)zRunEvalConfig.ExactMatchaxConfiguration for an exact match string evaluator. Parameters ---------- ignore_case : bool Whether to ignore case when comparing strings. ignore_punctuation : bool Whether to ignore punctuation when comparing strings. ignore_numbers : bool Whether to ignore numbers when comparing strings. rF ignore_caseignore_punctuationignore_numbersN) r#r$r%r&r EXACT_MATCHrr'rbrRrcrdrrrr! ExactMatch8s   rfc@s,eZdZUdZejZeed<dZe ed<dS)zRunEvalConfig.RegexMatchzConfiguration for a regex match string evaluator. Parameters ---------- flags : int The flags to pass to the regex. Example: re.IGNORECASE. rrflagsN) r#r$r%r&r REGEX_MATCHrr'rgintrrrr! RegexMatchJs rjc seZdZUdZejZeed<dZe e ed<dZ e e ed<dZ e eed<dZe eed<  d de e de eded dffd d ZZS) zRunEvalConfig.ScoreStringaConfiguration for a score string evaluator. This is like the criteria evaluator but it is configured by default to return a score on the scale from 1-10. It is recommended to normalize these scores by setting `normalize_by` to 10. Parameters ---------- criteria : Optional[CRITERIA_TYPE] The criteria to evaluate. llm : Optional[BaseLanguageModel] The language model to use for the evaluation chain. normalize_by: Optional[int] = None If you want to normalize the score, the denominator to use. If not provided, the score will be between 1 and 10 (by default). prompt : Optional[BasePromptTemplate] rNr<r= normalize_byrTrrc stjd||d|dS)N)r<rkrr@)rr<rkrr0rr!rAqsz"RunEvalConfig.ScoreString.__init__)NN)r#r$r%r&r SCORE_STRINGrr'r<rrr=r rkfloatrTr rrAr2rrr0r! ScoreStringVs$ rnc@seZdZUejZeed<dS)z RunEvalConfig.LabeledScoreStringrN)r#r$r%rLABELED_SCORE_STRINGrr'rrrr!LabeledScoreStringys rp)'r#r$r%r&rlistr5rrSINGLE_EVAL_CONFIG_TYPECUSTOM_EVALUATOR_TYPEr'r6rr7BATCH_EVALUATOR_LIKEr*r(r+r,r8r rrMr)rFrHrrrUrYr[r^rrarfrjrnrprrrr!r3]sN #   #r3N)/r&typingrrrrrrrlangchain_core.embeddingsr langchain_core.language_modelsr langchain_core.promptsr langsmithr langsmith.evaluation.evaluatorr rlangsmith.schemasrrpydanticrrr(langchain.evaluation.criteria.eval_chainr,langchain.evaluation.embedding_distance.baserrLlangchain.evaluation.schemarr)langchain.evaluation.string_distance.baserrQdictRUN_EVALUATOR_LIKErtrr)rsr(rrr3rrrr!s4$        $