fastreid.engine.defaults.DefaultTrainer Class Reference

Inheritance diagram for fastreid.engine.defaults.DefaultTrainer:

Public Member Functions
	__init__ (self, cfg)
	resume_or_load (self, resume=True)
	build_hooks (self)
	build_writers (self)
	train (self)
	build_model (cls, cfg)
	build_optimizer (cls, cfg, model)
	build_lr_scheduler (cls, cfg, optimizer)
	build_train_loader (cls, cfg)
	build_test_loader (cls, cfg, dataset_name)
	build_evaluator (cls, cfg, dataset_name, output_dir=None)
	test (cls, cfg, model)
Public Member Functions inherited from fastreid.engine.train_loop.SimpleTrainer
	run_step (self)
Public Member Functions inherited from fastreid.engine.train_loop.TrainerBase
	register_hooks (self, hooks)
	before_train (self)
	after_train (self)
	before_step (self)
	after_step (self)

Static Public Member Functions
	auto_scale_hyperparams (cfg, data_loader)

Public Attributes
	scheduler
	checkpointer
	start_iter
	max_iter
	cfg
	optimizer
	model
Public Attributes inherited from fastreid.engine.train_loop.SimpleTrainer
	model
	data_loader
	optimizer
	amp_enabled
	scaler
	iter
Public Attributes inherited from fastreid.engine.train_loop.TrainerBase
	iter
	max_iter

Protected Attributes
	_last_eval_results
Protected Attributes inherited from fastreid.engine.train_loop.SimpleTrainer
	_data_loader_iter
Protected Attributes inherited from fastreid.engine.train_loop.TrainerBase
	_hooks

Additional Inherited Members
Protected Member Functions inherited from fastreid.engine.train_loop.SimpleTrainer
	_detect_anomaly (self, losses, loss_dict)
	_write_metrics (self, dict metrics_dict)

Detailed Description

A trainer with default training logic. Compared to `SimpleTrainer`, it
contains the following logic in addition:
1. Create model, optimizer, scheduler, dataloader from the given config.
2. Load a checkpoint or `cfg.MODEL.WEIGHTS`, if exists.
3. Register a few common hooks.
It is created to simplify the **standard model training workflow** and reduce code boilerplate
for users who only need the standard training workflow, with standard features.
It means this class makes *many assumptions* about your training logic that
may easily become invalid in a new research. In fact, any assumptions beyond those made in the
:class:`SimpleTrainer` are too much for research.
The code of this class has been annotated about restrictive assumptions it mades.
When they do not work for you, you're encouraged to:
1. Overwrite methods of this class, OR:
2. Use :class:`SimpleTrainer`, which only does minimal SGD training and
   nothing else. You can then add your own hooks if needed. OR:
3. Write your own training loop similar to `tools/plain_train_net.py`.
Also note that the behavior of this class, like other functions/classes in
this file, is not stable, since it is meant to represent the "common default behavior".
It is only guaranteed to work well with the standard models and training workflow in fastreid.
To obtain more stable behavior, write your own training logic with other public APIs.
Attributes:
    scheduler:
    checkpointer:
    cfg (CfgNode):
Examples:
.. code-block:: python
    trainer = DefaultTrainer(cfg)
    trainer.resume_or_load()  # load last checkpoint or MODEL.WEIGHTS
    trainer.train()

Definition at line 167 of file defaults.py.

Constructor & Destructor Documentation

__init__()

fastreid.engine.defaults.DefaultTrainer.__init__	(		self,
			cfg )

Args:
    cfg (CfgNode):

Reimplemented from fastreid.engine.train_loop.SimpleTrainer.

Definition at line 200 of file defaults.py.

    def __init__(self, cfg):
        """
        Args:
            cfg (CfgNode):
        """
        logger = logging.getLogger("fastreid")
        if not logger.isEnabledFor(logging.INFO):  # setup_logger is not called for fastreid
            setup_logger()
 
        # Assume these objects must be constructed in this order.
        data_loader = self.build_train_loader(cfg)
        cfg = self.auto_scale_hyperparams(cfg, data_loader)
        model = self.build_model(cfg)
        optimizer = self.build_optimizer(cfg, model)
 
        # For training, wrap with DDP. But don't need this for inference.
        if comm.get_world_size() > 1:
            # ref to https://github.com/pytorch/pytorch/issues/22049 to set `find_unused_parameters=True`
            # for part of the parameters is not updated.
            model = DistributedDataParallel(
                model, device_ids=[comm.get_local_rank()], broadcast_buffers=False
            )
 
        super().__init__(model, data_loader, optimizer, cfg.SOLVER.AMP_ENABLED)
 
        self.scheduler = self.build_lr_scheduler(cfg, optimizer)
        # Assume no other objects need to be checkpointed.
        # We can later make it checkpoint the stateful hooks
        self.checkpointer = Checkpointer(
            # Assume you want to save checkpoints together with logs/statistics
            model,
            cfg.OUTPUT_DIR,
            save_to_disk=comm.is_main_process(),
            optimizer=optimizer,
            scheduler=self.scheduler,
        )
        self.start_iter = 0
        if cfg.SOLVER.SWA.ENABLED:
            self.max_iter = cfg.SOLVER.MAX_ITER + cfg.SOLVER.SWA.ITER
        else:
            self.max_iter = cfg.SOLVER.MAX_ITER
 
        self.cfg = cfg
 
        self.register_hooks(self.build_hooks())
 

Member Function Documentation

auto_scale_hyperparams()

fastreid.engine.defaults.DefaultTrainer.auto_scale_hyperparams ( cfg, data_loader )

static

    This is used for auto-computation actual training iterations,
    because some hyper-param, such as MAX_ITER, means training epochs rather than iters,
    so we need to convert specific hyper-param to training iterations.

Definition at line 466 of file defaults.py.

    def auto_scale_hyperparams(cfg, data_loader):
        r"""
        This is used for auto-computation actual training iterations,
        because some hyper-param, such as MAX_ITER, means training epochs rather than iters,
        so we need to convert specific hyper-param to training iterations.
        """
 
        cfg = cfg.clone()
        frozen = cfg.is_frozen()
        cfg.defrost()
 
        iters_per_epoch = len(data_loader.dataset) // cfg.SOLVER.IMS_PER_BATCH
        cfg.MODEL.HEADS.NUM_CLASSES = data_loader.dataset.num_classes
        cfg.SOLVER.MAX_ITER *= iters_per_epoch
        cfg.SOLVER.WARMUP_ITERS *= iters_per_epoch
        cfg.SOLVER.FREEZE_ITERS *= iters_per_epoch
        cfg.SOLVER.DELAY_ITERS *= iters_per_epoch
        for i in range(len(cfg.SOLVER.STEPS)):
            cfg.SOLVER.STEPS[i] *= iters_per_epoch
        cfg.SOLVER.SWA.ITER *= iters_per_epoch
        cfg.SOLVER.SWA.PERIOD *= iters_per_epoch
 
        ckpt_multiple = cfg.SOLVER.CHECKPOINT_PERIOD / cfg.TEST.EVAL_PERIOD
        # Evaluation period must be divided by 200 for writing into tensorboard.
        eval_num_mod = (200 - cfg.TEST.EVAL_PERIOD * iters_per_epoch) % 200
        cfg.TEST.EVAL_PERIOD = cfg.TEST.EVAL_PERIOD * iters_per_epoch + eval_num_mod
        # Change checkpoint saving period consistent with evaluation period.
        cfg.SOLVER.CHECKPOINT_PERIOD = int(cfg.TEST.EVAL_PERIOD * ckpt_multiple)
 
        logger = logging.getLogger(__name__)
        logger.info(
            f"Auto-scaling the config to num_classes={cfg.MODEL.HEADS.NUM_CLASSES}, "
            f"max_Iter={cfg.SOLVER.MAX_ITER}, wamrup_Iter={cfg.SOLVER.WARMUP_ITERS}, "
            f"freeze_Iter={cfg.SOLVER.FREEZE_ITERS}, delay_Iter={cfg.SOLVER.DELAY_ITERS}, "
            f"step_Iter={cfg.SOLVER.STEPS}, ckpt_Iter={cfg.SOLVER.CHECKPOINT_PERIOD}, "
            f"eval_Iter={cfg.TEST.EVAL_PERIOD}."
        )
 
        if frozen: cfg.freeze()
 
        return cfg

build_evaluator()

fastreid.engine.defaults.DefaultTrainer.build_evaluator	(		cls,
			cfg,
			dataset_name,
			output_dir = None )

Definition at line 424 of file defaults.py.

    def build_evaluator(cls, cfg, dataset_name, output_dir=None):
        data_loader, num_query = cls.build_test_loader(cfg, dataset_name)
        return data_loader, ReidEvaluator(cfg, num_query, output_dir)
 

build_hooks()

fastreid.engine.defaults.DefaultTrainer.build_hooks

(

self

)

Build a list of default hooks, including timing, evaluation,
checkpointing, lr scheduling, precise BN, writing events.
Returns:
    list[HookBase]:

Definition at line 262 of file defaults.py.

    def build_hooks(self):
        """
        Build a list of default hooks, including timing, evaluation,
        checkpointing, lr scheduling, precise BN, writing events.
        Returns:
            list[HookBase]:
        """
        logger = logging.getLogger(__name__)
        cfg = self.cfg.clone()
        cfg.defrost()
        cfg.DATALOADER.NUM_WORKERS = 0  # save some memory and time for PreciseBN
        cfg.DATASETS.NAMES = tuple([cfg.TEST.PRECISE_BN.DATASET])  # set dataset name for PreciseBN
 
        ret = [
            hooks.IterationTimer(),
            hooks.LRScheduler(self.optimizer, self.scheduler),
        ]
 
        if cfg.SOLVER.SWA.ENABLED:
            ret.append(
                hooks.SWA(
                    cfg.SOLVER.MAX_ITER,
                    cfg.SOLVER.SWA.PERIOD,
                    cfg.SOLVER.SWA.LR_FACTOR,
                    cfg.SOLVER.SWA.ETA_MIN_LR,
                    cfg.SOLVER.SWA.LR_SCHED,
                )
            )
 
        if cfg.TEST.PRECISE_BN.ENABLED and hooks.get_bn_modules(self.model):
            logger.info("Prepare precise BN dataset")
            ret.append(hooks.PreciseBN(
                # Run at the same freq as (but before) evaluation.
                self.model,
                # Build a new data loader to not affect training
                self.build_train_loader(cfg),
                cfg.TEST.PRECISE_BN.NUM_ITER,
            ))
 
        if cfg.MODEL.FREEZE_LAYERS != [''] and cfg.SOLVER.FREEZE_ITERS > 0:
            freeze_layers = ",".join(cfg.MODEL.FREEZE_LAYERS)
            logger.info(f'Freeze layer group "{freeze_layers}" training for {cfg.SOLVER.FREEZE_ITERS:d} iterations')
            ret.append(hooks.FreezeLayer(
                self.model,
                self.optimizer,
                cfg.MODEL.FREEZE_LAYERS,
                cfg.SOLVER.FREEZE_ITERS,
            ))
        # Do PreciseBN before checkpointer, because it updates the model and need to
        # be saved by checkpointer.
        # This is not always the best: if checkpointing has a different frequency,
        # some checkpoints may have more precise statistics than others.
        if comm.is_main_process():
            ret.append(hooks.PeriodicCheckpointer(self.checkpointer, cfg.SOLVER.CHECKPOINT_PERIOD))
 
        def test_and_save_results():
            self._last_eval_results = self.test(self.cfg, self.model)
            return self._last_eval_results
 
        # Do evaluation after checkpointer, because then if it fails,
        # we can use the saved checkpoint to debug.
        ret.append(hooks.EvalHook(cfg.TEST.EVAL_PERIOD, test_and_save_results))
 
        if comm.is_main_process():
            # run writers in the end, so that evaluation metrics are written
            ret.append(hooks.PeriodicWriter(self.build_writers(), 200))
 
        return ret
 

build_lr_scheduler()

fastreid.engine.defaults.DefaultTrainer.build_lr_scheduler	(		cls,
			cfg,
			optimizer )

It now calls :func:`fastreid.solver.build_lr_scheduler`.
Overwrite it if you'd like a different scheduler.

Definition at line 394 of file defaults.py.

    def build_lr_scheduler(cls, cfg, optimizer):
        """
        It now calls :func:`fastreid.solver.build_lr_scheduler`.
        Overwrite it if you'd like a different scheduler.
        """
        return build_lr_scheduler(cfg, optimizer)
 

build_model()

fastreid.engine.defaults.DefaultTrainer.build_model	(		cls,
			cfg )

Returns:
    torch.nn.Module:
It now calls :func:`fastreid.modeling.build_model`.
Overwrite it if you'd like a different model.

Definition at line 371 of file defaults.py.

    def build_model(cls, cfg):
        """
        Returns:
            torch.nn.Module:
        It now calls :func:`fastreid.modeling.build_model`.
        Overwrite it if you'd like a different model.
        """
        model = build_model(cfg)
        # logger = logging.getLogger(__name__)
        # logger.info("Model:\n{}".format(model))
        return model
 

build_optimizer()

fastreid.engine.defaults.DefaultTrainer.build_optimizer	(		cls,
			cfg,
			model )

Returns:
    torch.optim.Optimizer:
It now calls :func:`fastreid.solver.build_optimizer`.
Overwrite it if you'd like a different optimizer.

Definition at line 384 of file defaults.py.

    def build_optimizer(cls, cfg, model):
        """
        Returns:
            torch.optim.Optimizer:
        It now calls :func:`fastreid.solver.build_optimizer`.
        Overwrite it if you'd like a different optimizer.
        """
        return build_optimizer(cfg, model)
 

build_test_loader()

fastreid.engine.defaults.DefaultTrainer.build_test_loader	(		cls,
			cfg,
			dataset_name )

Returns:
    iterable
It now calls :func:`fastreid.data.build_detection_test_loader`.
Overwrite it if you'd like a different data loader.

Definition at line 414 of file defaults.py.

    def build_test_loader(cls, cfg, dataset_name):
        """
        Returns:
            iterable
        It now calls :func:`fastreid.data.build_detection_test_loader`.
        Overwrite it if you'd like a different data loader.
        """
        return build_reid_test_loader(cfg, dataset_name)
 

build_train_loader()

fastreid.engine.defaults.DefaultTrainer.build_train_loader	(		cls,
			cfg )

Returns:
    iterable
It now calls :func:`fastreid.data.build_detection_train_loader`.
Overwrite it if you'd like a different data loader.

Definition at line 402 of file defaults.py.

    def build_train_loader(cls, cfg):
        """
        Returns:
            iterable
        It now calls :func:`fastreid.data.build_detection_train_loader`.
        Overwrite it if you'd like a different data loader.
        """
        logger = logging.getLogger(__name__)
        logger.info("Prepare training set")
        return build_reid_train_loader(cfg)
 

build_writers()

fastreid.engine.defaults.DefaultTrainer.build_writers

(

self

)

Build a list of writers to be used. By default it contains
writers that write metrics to the screen,
a json file, and a tensorboard event file respectively.
If you'd like a different list of writers, you can overwrite it in
your trainer.
Returns:
    list[EventWriter]: a list of :class:`EventWriter` objects.
It is now implemented by:
.. code-block:: python
    return [
        CommonMetricPrinter(self.max_iter),
        JSONWriter(os.path.join(self.cfg.OUTPUT_DIR, "metrics.json")),
        TensorboardXWriter(self.cfg.OUTPUT_DIR),
    ]

Definition at line 331 of file defaults.py.

    def build_writers(self):
        """
        Build a list of writers to be used. By default it contains
        writers that write metrics to the screen,
        a json file, and a tensorboard event file respectively.
        If you'd like a different list of writers, you can overwrite it in
        your trainer.
        Returns:
            list[EventWriter]: a list of :class:`EventWriter` objects.
        It is now implemented by:
        .. code-block:: python
            return [
                CommonMetricPrinter(self.max_iter),
                JSONWriter(os.path.join(self.cfg.OUTPUT_DIR, "metrics.json")),
                TensorboardXWriter(self.cfg.OUTPUT_DIR),
            ]
        """
        # Assume the default print/log frequency.
        return [
            # It may not always print what you want to see, since it prints "common" metrics only.
            CommonMetricPrinter(self.max_iter),
            JSONWriter(os.path.join(self.cfg.OUTPUT_DIR, "metrics.json")),
            TensorboardXWriter(self.cfg.OUTPUT_DIR),
        ]
 

resume_or_load()

fastreid.engine.defaults.DefaultTrainer.resume_or_load	(		self,
			resume = True )

If `resume==True`, and last checkpoint exists, resume from it.
Otherwise, load a model specified by the config.
Args:
    resume (bool): whether to do resume or not

Definition at line 246 of file defaults.py.

    def resume_or_load(self, resume=True):
        """
        If `resume==True`, and last checkpoint exists, resume from it.
        Otherwise, load a model specified by the config.
        Args:
            resume (bool): whether to do resume or not
        """
        # The checkpoint stores the training iteration that just finished, thus we start
        # at the next iteration (or iter zero if there's no checkpoint).
        checkpoint = self.checkpointer.resume_or_load(self.cfg.MODEL.WEIGHTS, resume=resume)
 
        if resume and self.checkpointer.has_checkpoint():
            self.start_iter = checkpoint.get("iteration", -1) + 1
            # The checkpoint stores the training iteration that just finished, thus we start
            # at the next iteration (or iter zero if there's no checkpoint).
 

test()

fastreid.engine.defaults.DefaultTrainer.test	(		cls,
			cfg,
			model )

Args:
    cfg (CfgNode):
    model (nn.Module):
Returns:
    dict: a dict of result metrics

Definition at line 429 of file defaults.py.

    def test(cls, cfg, model):
        """
        Args:
            cfg (CfgNode):
            model (nn.Module):
        Returns:
            dict: a dict of result metrics
        """
        logger = logging.getLogger(__name__)
 
        results = OrderedDict()
        for idx, dataset_name in enumerate(cfg.DATASETS.TESTS):
            logger.info("Prepare testing set")
            try:
                data_loader, evaluator = cls.build_evaluator(cfg, dataset_name)
            except NotImplementedError:
                logger.warn(
                    "No evaluator found. implement its `build_evaluator` method."
                )
                results[dataset_name] = {}
                continue
            results_i = inference_on_dataset(model, data_loader, evaluator)
            results[dataset_name] = results_i
 
        if comm.is_main_process():
            assert isinstance(
                results, dict
            ), "Evaluator must return a dict on the main process. Got {} instead.".format(
                results
            )
            print_csv_format(results)
 
        if len(results) == 1: results = list(results.values())[0]
 
        return results
 

train()

fastreid.engine.defaults.DefaultTrainer.train

(

self

)

Run training.
Returns:
    OrderedDict of results, if evaluation is enabled. Otherwise None.

Reimplemented from fastreid.engine.train_loop.TrainerBase.

Definition at line 356 of file defaults.py.

    def train(self):
        """
        Run training.
        Returns:
            OrderedDict of results, if evaluation is enabled. Otherwise None.
        """
        super().train(self.start_iter, self.max_iter)
        if comm.is_main_process():
            assert hasattr(
                self, "_last_eval_results"
            ), "No evaluation results obtained during training!"
            # verify_results(self.cfg, self._last_eval_results)
            return self._last_eval_results
 

Member Data Documentation

_last_eval_results

fastreid.engine.defaults.DefaultTrainer._last_eval_results

protected

Definition at line 318 of file defaults.py.

cfg

fastreid.engine.defaults.DefaultTrainer.cfg

Definition at line 242 of file defaults.py.

checkpointer

fastreid.engine.defaults.DefaultTrainer.checkpointer

Definition at line 228 of file defaults.py.

max_iter

fastreid.engine.defaults.DefaultTrainer.max_iter

Definition at line 238 of file defaults.py.

model

fastreid.engine.defaults.DefaultTrainer.model

Definition at line 291 of file defaults.py.

optimizer

fastreid.engine.defaults.DefaultTrainer.optimizer

Definition at line 277 of file defaults.py.

scheduler

fastreid.engine.defaults.DefaultTrainer.scheduler

Definition at line 225 of file defaults.py.

start_iter

fastreid.engine.defaults.DefaultTrainer.start_iter

Definition at line 236 of file defaults.py.

---

The documentation for this class was generated from the following file:

• smreid/fastreid/engine/defaults.py