python 2.7 - Scons AttributeError: 'builtin_function_or_method' object has no attribute 'dispatch' -
i have sconstruct script instantiating object.this object internally calls method run multiprocessing module. example shown below
this object before calling function unpickles file , pass inputs multiprocessing module.
def run_scons(self,inpfile,outfile): # unpickle input parameter fid=open(inpfile,'rb') input_data=pkls.load(fid) my_results=[] #run solver in loop my_data in input_data: work_ers=len(my_data) pool = pool(processes=work_ers) a_result=pool.map_async(my_solver, my_data) pool.close() pool.join() my_results.append(a_result.get()) fid.close() fid_out=open(outfile,'wb+') pkls.dump(rot_full_results,fid_out)
i following error when executing same function via scons.
pool = pool(processes=work_ers) file "c:\python27\lib\multiprocessing\__init__.py", line 232, in pool return pool(processes, initializer, initargs, maxtasksperchild) file "c:\python27\lib\multiprocessing\pool.py", line 138, in __init__ self._setup_queues() file "c:\python27\lib\multiprocessing\pool.py", line 232, in _setup_queues .queues import simplequeue file "c:\python27\lib\multiprocessing\queues.py", line 48, in <module> multiprocessing.synchronize import lock, boundedsemaphore, semaphore, condition file "c:\python27\lib\multiprocessing\synchronize.py", line 48, in <module> multiprocessing.forking import assert_spawning, popen file "c:\python27\lib\multiprocessing\forking.py", line 60, in <module> class forkingpickler(pickler): file "c:\python27\lib\multiprocessing\forking.py", line 61, in forkingpickler dispatch = pickler.dispatch.copy() attributeerror: 'builtin_function_or_method' object has no attribute 'dispatch' scons: building terminated because of errors.
after reading error, found out scons has hack around renames pickle module cpickle, whereas multiprocessing module looking cpickle , goes down. there way around this?
i using multiprocessing process 'job' files in parallel. doing in sconstruct file:
if getoption("run_jobs"): my_jobs = # code produces list of job objects ecode = build_support.mp_run_jobs(my_jobs) exit(ecode)
but ran exception above:
attributeerror: 'builtin_function_or_method' object has no attribute 'dispatch'
on linux using 'scons_horrible_regression_test_hack' works:
$ scons_horrible_regression_test_hack=1 scons --run-jobs
but don't want have export symbol on linux , windows , mac etc.
i did find workaround, delete pickle modules , re-import them this:
if getoption("run_jobs"): # workaround scons.compat module renaming import imp del sys.modules['pickle'] del sys.modules['cpickle'] sys.modules['pickle'] = imp.load_module('pickle', *imp.find_module('pickle')) sys.modules['cpickle'] = imp.load_module('cpickle', *imp.find_module('cpickle')) import pickle import cpickle print "(pickle == cpickle) = ", (pickle == cpickle) my_jobs = # code produces list of job objects ecode = build_support.mp_run_jobs(my_jobs) exit(ecode)
now multiprocessing of jobs works expected!
Comments
Post a Comment