Neural Architecture Search (NAS)¶
Goal¶
To automatically search a network architecture that leads to the best accuracy.
Architecture Options¶
Blocks: residual block , inception block, bottleneck block, etc.
Layers: convs, pooling, fc, etc.
Hyperparameters: number of filters, size of kernel, stride, padding, etc.
Search space¶
The set containing all the possible architectures
Example of constructing search space
Warning: Need to import nni.retiarii.nn.pytorch as nn to run the following code
self.layer = nn.LayerChoice([
ops.PoolBN('max', channels, 3, stride, 1),
ops.SepConv(channels, channels, 3, stride, 1),
nn.Identity()
])
The search space visualization in this experiment
For more details, please refer to NNI
Oneshot vs Multi-trail NAS¶
Multi-trail NAS: In Multi-trail NAS, users need model evaluator to evaluate the performance of each sampled model, and need an exploration strategy to sample models from a defined model space. Multi-trail mechanism is easy to understand and implement, therefore, implementing new functions (ex: efficiency limits) is much easier in Multi-trial. However, Multi-trail also needs more traininig time than Oneshot.
Oneshot NAS: One-shot NAS algorithms leverage weight sharing among models in neural architecture search space to train a supernet, and use this supernet to guide the selection of better models. This type of algorihtms greatly reduces computational resource compared to independently training each model from scratch (Multi-trial NAS). The following figure shows how Oneshot NAS trains a supernet.
Pros and Cons¶
Pros:
NAS can significantly increase the accuracy of models.
Multi-trail NAS can even add time limits to ensure both accuracy and efficiency of models.
Cons:
The training time for NAS is extremely long (especially Multi-trail NAS).