pickle - Frequently Updating Stored Data for a Numerical Experiment using Python -


i running numerical experiment requires many iterations. after each iteration, store data in pickle file or pickle-like file in case program times-out or data structure becomes tapped. best way proceed. here skeleton code:

data_dict = {}                       # maybe dictionary not best choice j in parameters:                 # j = (alpha, beta, gamma) , cycle through     k in number_of_experiments:  # lots of experiments (10^4)         file = open('storage.pkl', 'ab')         data = experiment()          # experiment returns numerical value                                      # experiment takes ~ 1 seconds, increase                                      # parameters scale         data_dict.setdefault(j, []).append(data)         pickle.dump(data_dict, file)         file.close()

questions:

  1. is shelve better choice here? or other python library not aware?
  2. i using data dict because it's easier code , more flexible if need change things more experiments. huge advantage use pre-allocated array?
  3. does opening , closing files affect run time? can check on progress in addition text logs have set up.

thank help!

  1. assuming using numpy numerical experiments, instead of pickle suggest using numpy.savez.
  2. keep simple , make optimizations if feel script runs long.
  3. opening , closing files affect run time, having backup anyway better.

and use collections.defaultdict(list) instead of plain dict , setdefault.


Comments

Post a Comment

Popular posts from this blog

javascript - RequestAnimationFrame not working when exiting fullscreen switching space on Safari -

request animation frame stops working when exiting fullscreen clicking on original space's safari window. fine if fullscreen mode canceled escape key or calling cancelfullscreen(). steps reproduce: open https://dl.dropboxusercontent.com/u/769042/prezi/safari-fullscreen.html click "draw raf", kittie appears click "fullscreen", go fullscreen click "draw raf", kittie appears go space original safari was, showing "click exit fullscreen mode", click anywhere, out fullscreen click "draw raf", nothing happens what handling click calling: window.requestanimationframe(draw); which draws on canvas context: function draw() { ctx.drawimage(img, math.random()*500|0, math.random()*400|0, 100, 100); } i checked .hidden , .visibilitystate, updated correctly. tested on osx 10.9.3, safari 7.0.4 (9537.76.4). has workaround/solution other switching old settimeout? this sounds webkit bug 88940 : using reque
Read more

Python ctypes access violation with const pointer arguments -

i have api i'm trying wrap in python (2.7.6 on win7) code using ctypes. here's api: client_dllfunc bool clientapi search_exporttoclipcopy(clienthsearch handle, int channel, lpctstr filename, const time_t& from, const time_t& to, const bool* cameras, int length, bool usepassword, lpctstr password, bool includetextin = false, bool excludeplayer = false); the issue i'm having cameras argument; , i've had issue other const pointer arguments well. here's how i'm wrapping api: def search_exporttoclipcopy(self, hsearch, csearch, filename, tfrom, tto, cameras, length, usepassword, password, includetextin=true, excludeplayer=false): exporttoclipcopy = self.sdkdll.search_exporttoclipcopy exporttoclipcopy.argtypes = [c_int, c_int, c_wchar_p, c_long, c_long, pointer(c_bool), c_int, c_bool, c_wchar_p, c_bool, c_bool] exporttoclipcopy.restype = c_bool exporttoclipcopy(hsearch, csearch, filename, tfrom, tto, cameras, length, usepassword, pass
Read more