[NLPL Task Force (A)] rolling your own BERT (and maybe ELMo) on Saga

[Antti Virtanen]

I think this is a good move. While you could probably fit enough examples per batch by using 2 or 3 times the cards, the half-precision performance of the V100s is a good deal better. Plus Puhti is a system we know we can run this on.

-Antti

From: Stephan Oepen
Sent: perjantai 31. tammikuuta 2020 19.53
To: Antti Virtanen
Cc: Andrei Kutuzov; infrastructure; outreach at nlpl.eu
Subject: Re: rolling your own BERT (and maybe ELMo) on Saga

belatedly, thanks for the link, antti!

> Here's a (quick and dirty) repo for the code we used to train FinBERT: https://github.com/haamis/DeepLearningExamples_FinBERT/tree/master/TensorFlow/LanguageModeling/BERT_nonscaling. This one has the sbatch files used: https://github.com/haamis/BERT-pretraining

the README claims that V100 gpus are required, whereas Saga only has
P100 cards.  so i just gave up on the prospect of getting this to run
in norway and rather requested an allocation of billing units on Puhti
:-).  i think for your presentation next week, you can just assume
that Saga is not a relevant target system for this work.

looking forward to the tutorial!  oe

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.nlpl.eu/archives/infrastructure/attachments/20200131/b5d394ae/attachment.htm>