o Jjg«ã@sødZddlmZddlmZmZddlmZmZm Z ddl m Z m Z m Z ddlmZddlmZmZmZddlmZmZdd lmZdd lmZdd lmZmZmZdd lm Z dd l!m"Z"m#Z#m$Z$m%Z%ddl&m'Z'm(Z(ddl)m*Z*m+Z+m,Z,gd¢Z-dS)aº **Evaluation** chains for grading LLM and Chain outputs. This module contains off-the-shelf evaluation chains for grading the output of LangChain primitives such as language models and chains. **Loading an evaluator** To load an evaluator, you can use the :func:`load_evaluators ` or :func:`load_evaluator ` functions with the names of the evaluators to load. .. code-block:: python from langchain.evaluation import load_evaluator evaluator = load_evaluator("qa") evaluator.evaluate_strings( prediction="We sold more than 40,000 units last week", input="How many units did we sell last week?", reference="We sold 32,378 units", ) The evaluator must be one of :class:`EvaluatorType `. **Datasets** To load one of the LangChain HuggingFace datasets, you can use the :func:`load_dataset ` function with the name of the dataset to load. .. code-block:: python from langchain.evaluation import load_dataset ds = load_dataset("llm-math") **Some common use cases for evaluation include:** - Grading the accuracy of a response against ground truth answers: :class:`QAEvalChain ` - Comparing the output of two models: :class:`PairwiseStringEvalChain ` or :class:`LabeledPairwiseStringEvalChain ` when there is additionally a reference label. - Judging the efficacy of an agent's tool usage: :class:`TrajectoryEvalChain ` - Checking whether an output complies with a set of criteria: :class:`CriteriaEvalChain ` or :class:`LabeledCriteriaEvalChain ` when there is additionally a reference label. - Computing semantic difference between a prediction and reference: :class:`EmbeddingDistanceEvalChain ` or between two predictions: :class:`PairwiseEmbeddingDistanceEvalChain ` - Measuring the string distance between a prediction and reference :class:`StringDistanceEvalChain ` or between two predictions :class:`PairwiseStringDistanceEvalChain ` **Low-level API** These evaluators implement one of the following interfaces: - :class:`StringEvaluator `: Evaluate a prediction string against a reference label and/or input context. - :class:`PairwiseStringEvaluator `: Evaluate two prediction strings against each other. Useful for scoring preferences, measuring similarity between two chain or llm agents, or comparing outputs on similar inputs. - :class:`AgentTrajectoryEvaluator ` Evaluate the full sequence of actions taken by an agent. These interfaces enable easier composability and usage within a higher level evaluation framework. é)ÚTrajectoryEvalChain)ÚLabeledPairwiseStringEvalChainÚPairwiseStringEvalChain)ÚCriteriaÚCriteriaEvalChainÚLabeledCriteriaEvalChain)ÚEmbeddingDistanceÚEmbeddingDistanceEvalChainÚ"PairwiseEmbeddingDistanceEvalChain)ÚExactMatchStringEvaluator)Ú load_datasetÚload_evaluatorÚload_evaluators)ÚJsonEqualityEvaluatorÚJsonValidityEvaluator)ÚJsonEditDistanceEvaluator)ÚJsonSchemaEvaluator)ÚContextQAEvalChainÚCotQAEvalChainÚ QAEvalChain)ÚRegexMatchStringEvaluator)ÚAgentTrajectoryEvaluatorÚ EvaluatorTypeÚPairwiseStringEvaluatorÚStringEvaluator)ÚLabeledScoreStringEvalChainÚScoreStringEvalChain)ÚPairwiseStringDistanceEvalChainÚStringDistanceÚStringDistanceEvalChain)rr rrrrrrrrrrrrr r rrrrrr r rrrrrrrN).Ú__doc__Úlangchain.evaluation.agentsrÚlangchain.evaluation.comparisonrrÚlangchain.evaluation.criteriarrrÚ'langchain.evaluation.embedding_distancerr r Ú%langchain.evaluation.exact_match.baser Úlangchain.evaluation.loadingr r rÚ!langchain.evaluation.parsing.baserrÚ*langchain.evaluation.parsing.json_distancerÚ(langchain.evaluation.parsing.json_schemarÚlangchain.evaluation.qarrrÚ%langchain.evaluation.regex_match.baserÚlangchain.evaluation.schemarrrrÚlangchain.evaluation.scoringrrÚ$langchain.evaluation.string_distancerrrÚ__all__©r0r0úU/var/www/html/zoom/venv/lib/python3.10/site-packages/langchain/evaluation/__init__.pyÚs  7