Python: subprocess memory leak -
i want run serial program on multiple cores @ same time , need multiple time (in loop). use subprocess.popen distribute jobs on processors limiting number of jobs number of available processors. add jobs list , check poll() if jobs done, remove them list , continue submission until total number of jobs completed.
i have been looking on web , found couple of interesting scripts , came out adapted version:
nextproc = 0 processes = [] while (len(processes) < limitproc): # here assume limitproc < ncores input = filelist[nextproc]+'.in' # filelist: list of input file output = filelist[nextproc]+'.out' # list of output file cwd = pathlist[nextproc] # list of paths processes.append(subprocess.popen(['myprogram','-i',input,'-screen',output],cwd=cwd,bufsize=-1)) nextproc += 1 time.sleep(wait) while (len(processes) > 0): # loop until processes done time.sleep(wait) in xrange(len(processes)-1, -1, -1): # remove processes done (traverse backward) if processes[i].poll() not none: del processes[i] time.sleep(wait) while (len(processes) < limitproc) , (nextproc < maxprocesses): # submit new processes output = filelist[nextproc]+'.out' input = filelist[nextproc]+'.in' cwd = pathlist[nextproc] processes.append(subprocess.popen(['myprogram','-i',input,'-screen',output],cwd=cwd,bufsize=-1)) nextproc += 1 time.sleep(wait) print 'jobs done'
i run script in loop , problem execution time increases 1 step another. here graph: http://i62.tinypic.com/2lk8f41.png
myprogram time execution constant. i'd glad if explain me causing leak.
thanks lot, begbi
Comments
Post a Comment