Fix: NotebookProgressCallback crash when evaluating with the Trainer#44949
Open
Charly21r wants to merge 3 commits intohuggingface:mainfrom
Open
Fix: NotebookProgressCallback crash when evaluating with the Trainer#44949Charly21r wants to merge 3 commits intohuggingface:mainfrom
Charly21r wants to merge 3 commits intohuggingface:mainfrom
Conversation
e61e591 to
e25f2a6
Compare
Member
|
does this fix your issue @HenrikEilers ? |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Fixes #44936
This PR fixes an issue with
NotebookProgressCallbackin theTrainerwhere calling evaluate() before or after training would crash due to the training tracker beingNone. The callback now properly handles evaluation even if training has not yet started or if it has already finished, ensuring metrics can be computed and displayed.Previously, the
on_evaluatemethod assumed thatself.training_trackerwas always initialized, but:self.training_trackerhas not being initialised byon_train_beginyet.on_train_endsetsself.training_trackertoNone, so callingon_evaluateafterwards would fail.Fix:
on_evaluatenow checks whether self.training_tracker exists before using it, and safely handles cases where it isNone. This prevents crashes and ensures evaluation can run regardless of training state.Additionally, new unit tests were added to ensure that evaluation works in this scenario, and existing notebook callback tests were updated to cover this case. This improves robustness of notebook-based workflows, especially in Jupyter or Colab environments.
Code Agent Policy
Before submitting
Who can review?
@SunMarc
Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.