Use evaluation loop #1543
Use evaluation loop #1543
Conversation
src/pykeen/pipeline/api.py
Outdated
| model=model_instance, | ||
| triples_factory=evaluation_factory, | ||
| evaluator=evaluator_instance, | ||
| # TODO: mode support? |
There was a problem hiding this comment.
which mode is already considered the "default"?
maybe just make this an argument, and raise a NotImplementedError if it's set to anything else (for now)
There was a problem hiding this comment.
None is the default (which corresponds to the transductive setting).
There was a problem hiding this comment.
on second look, the mode is passed through to the evaluator instance, so the way to support this is during construction of that
| """ | ||
| self.model = model | ||
| self.evaluator = evaluator | ||
| self.dataset = dataset | ||
| if mode is not None: | ||
| raise NotImplementedError(f"non-transductive evaluation mode is not yet implemented: {mode}") |
There was a problem hiding this comment.
The evaluation loop supports it. The only issue is that we don't expose the mode through the pipeline. However, this may be acceptable since adding support for inductive models to the pipeline could make it more difficult to determine which parameter combinations to provide.
There was a problem hiding this comment.
thanks! I noticed I misunderstood this and updated it after the first commit.
There was definitely some discussion somewhere about how to document exposing the inductive mode via the pipeline (#1503 (comment), moved to its own issue in #1544)
I wouldn't block up this PR with solving that problem
| @@ -169,12 +175,16 @@ def evaluate( | |||
| batch_size = determine_maximum_batch_size( | |||
| batch_size=batch_size, device=self.model.device, maximum_batch_size=len(self.dataset) | |||
| ) | |||
| # set upper limit for slice size | |||
| # TODO: if we knew the targets here, we could guess this better | |||
| slice_size = max(self.model.num_entities, self.model.num_relations) | |||
There was a problem hiding this comment.
there are parts of the pipeline that actually try to pass slice_size into this function (I am doing an experiment with typeddicts locally, which would make this PR much messier, but I found this)
so maybe slice_size should be an optional[int] where this gets calculated if it's not passed directly (or does this blow up AMO?)
There was a problem hiding this comment.
Update the code to use
EvaluationLoopinstead of theEvaluator.evaluate.EvaluationLoopuses a PyTorch dataset and dataloader to improve communication and computation overlap. In the filtered setting,EvaluationLoopprecomputes the filter positions once and re-uses them next time.Also implement support for slicing.