Click For Photo: https://regmedia.co.uk/2016/07/29/milk_in_teapot_shock.jpg
The shortest time for training a neural network using the popular ImageNet dataset has been slashed again, it is claimed, from the previously held record of four minutes to just one and a half.
Training is arguably the most important and tedious part of deep learning. A small mountain of data to teach a neural network to identify things, or otherwise make decisions from future inputs, is fed into the software, and filtered through multiple layers of intense matrix math calculations to train it. Developers long for snappy turnarounds in the order of minutes, rather than hours or days of waiting, so they can tweak their models for optimum performance, and test them, before the systems can be deployed.
Reeducation - Sessions - Facial-recognition - Voice-recognition - Systems
Shorter reeducation sessions also means facial-recognition, voice-recognition, and similar systems can be rapidly updated, tweaked, or improved on-the-fly.
There are all sorts of tricks to shave off training times. A common tactic is to run through the dataset quickly by increasing the batch size so that the model processes more samples per iteration. It decreases the overall accuracy, however, so it’s a bit of a balancing act.
Tactic - Mix - Half-precision - Floating - Point
Another tactic is to use a mix of half-precision floating point, aka FP16, as well as single-precision, FP32. This, for one thing, alleviates the memory bandwidth pressure on the GPUs or whatever chips you're using to accelerate the machine-learning math in hardware, though you may face some loss of accuracy.
Researchers at SenseTime, a Hong Kong-based computer-vision startup valued over $1bn, and Nanyang Technological University in Singapore, say they used these techniques to train AlexNet, an image-recognition convolutional neural network, on ImageNet in just 1.5 minutes albeit it with a 58.2 per cent accuracy.
Nvidia - Tesla - Volta - V100
It required 512 of Nvidia’s 2017-era Tesla Volta V100...
Wake Up To Breaking News!