[NLPL Task Force (A)] big array job

[Asad Sayeed]

Hi,

The abel documentation says users are allowed to run up to 400 jobs 
simultaneously.  If I run arrayrun 4x on different segments of the 
corpus, will I get myself into trouble with the authorities or 
something?  400 at a time is a significant time saving for me, obviously 
(2.5 days for the whole thing).

Thanks.

Yours,
--Asad.


On 2019-02-16 01:29 PM, Asad Sayeed wrote:
> Hi Stephan,
>
> I am now trying to scale up my SRL task "for real" over 70M sentences, 
> divided up into 3500 segments/tasks, each taking about 12G memory each 
> and taking about 7 hours.  I am trying to use arrayrun on abel on my 
> script.  However, it seems like arrayrun will only activate 100 jobs 
> at a time.  This will take 10 days to run the entire job, which is 
> slower than the much smaller cluster I was running it on elsewhere 
> (where I can run about 300 at a time and take 14 hours, for about 7 
> days).  I was hoping to gain a signficant turnaround time for 
> experimentation on abel.   Is there any way to get more on abel or is 
> that a hard limit?
>
> Thanks.
>
> Yours,
> --Asad.
>