python multithreading and file locking issues -
i have implemented multithreaded code in 2 ways, in both ways got error. explain causes problem?
in version 1, got exception saying 2 arguments passed writekey function instead of one. in version 2, 1 of threads reads empty line, therefore exception raised while processing empty string.
i using locks, shouldn't prevent multiple threads accessing function or file @ same time?
version 1:
class somethread(threading.thread): def __init__(self, somequeue, lockfile): threading.thread.__init__(self) self.myqueue = somequeue self.myfilelock = lockfile def writekey(key): if os.path.exists(os.path.join('.', outfile)): open(outfile, 'r') fc: readkey = int(fc.readline().rstrip()) os.remove(os.path.join('.', outfile)) open(outfile, 'w') fw: if readkey > key: fw.write(str(readkey)) else: fw.write(str(key)) def run(self): while(true): dict = self.myqueue.get() self.myfilelock.acquire() try: self.writekey(dict.get("key")) finally: self.myfilelock.release() self.myqueue.task_done() populatequeue() # populate queue objects filelock = threading.lock() in range(threadnum): thread = somethread(somequeue, filelock) thread.setdaemon(true) thread.start() somequeue.join()
version 2:
def writekey(key): if os.path.exists(os.path.join('.', outfile)): open(outfile, 'r') fc: # something... os.remove(os.path.join('.', outfile)) open(outfile, 'w') fw: # something... class somethread(threading.thread): def __init__(self, somequeue, lockfile): threading.thread.__init__(self) self.myqueue = somequeue self.myfilelock = lockfile def run(self): while(true): dict = self.myqueue.get() self.myfilelock.acquire() try: writekey(dict.get("key")) finally: myfilelock.release() self.myqueue.task_done() # same above ....
in version 1, def writekey(key)
should declared "self" first parameter, i.e.
def writekey(self, key):
the problem in version 2 less clear. assume empty line being read while reading outfile
. normal , indicates end-of-file has been reached. break out of read loop. preferable read file line-by-line in for
loop, e.g.
with open(outfile, 'r') fc: line in fc: # process line
the loop terminate naturally upon reaching end-of-file.
Comments
Post a Comment