symfony - Populating FosElasticaBundle running out of php memory, possible memory leak? -


i've installed foselasticabundle , have working cross section of data.

my problem arises in have 14m rows need use build index. ran populate command , after 6 hours yesterday errored out @ 10.8% memory error:

php fatal error:  allowed memory size of 2147483648 bytes exhausted (tried allocate 52277 bytes) in /var/www/html/vendor/monolog/monolog/src/monolog/formatter/lineformatter.php on line 111

as can see i've set php memory limit 2g should quite excessive.

the last line before error looked like

populating index/entity, 10.8% (1315300/12186320), 36 objects/s (ram : current=2045mo peak=2047mo)

and current , peak ticking every line, starting around 30mb.

my assumption here there sort of memory leak? surely php's memory shouldn't exhausted process. i've tried command parameters

app/console fos:elastica:populate --no-debug --no-reset --env=prod

but watch running current memory still ticking up.

any thoughts on might going on here , can debug it? found discussion sounds problem, doesn't present solution: https://github.com/friendsofsymfony/foselasticabundle/issues/82. i'm using doctrine , default provider.

thank you-

i'm not able solve memory leak entirely, running command 

app/console fos:elastica:populate --no-debug --no-reset --env=prod --offset=n

i've been able populate in batches. drastically cut down amount of memory leaking turning off logger, using solution on page

https://github.com/friendsofsymfony/foselasticabundle/issues/273

setting php memory_limit 4g (!) i'm able more 5m records populated without error, , after couple of batches should done process.

most solutions seem involve writing custom provider (see https://github.com/friendsofsymfony/foselasticabundle/issues/457) through ridiculous memory_limit , limiting memory leak as possible didn't need to.


Comments

Post a Comment

Popular posts from this blog

javascript - RequestAnimationFrame not working when exiting fullscreen switching space on Safari -

request animation frame stops working when exiting fullscreen clicking on original space's safari window. fine if fullscreen mode canceled escape key or calling cancelfullscreen(). steps reproduce: open https://dl.dropboxusercontent.com/u/769042/prezi/safari-fullscreen.html click "draw raf", kittie appears click "fullscreen", go fullscreen click "draw raf", kittie appears go space original safari was, showing "click exit fullscreen mode", click anywhere, out fullscreen click "draw raf", nothing happens what handling click calling: window.requestanimationframe(draw); which draws on canvas context: function draw() { ctx.drawimage(img, math.random()*500|0, math.random()*400|0, 100, 100); } i checked .hidden , .visibilitystate, updated correctly. tested on osx 10.9.3, safari 7.0.4 (9537.76.4). has workaround/solution other switching old settimeout? this sounds webkit bug 88940 : using reque
Read more

Python ctypes access violation with const pointer arguments -

i have api i'm trying wrap in python (2.7.6 on win7) code using ctypes. here's api: client_dllfunc bool clientapi search_exporttoclipcopy(clienthsearch handle, int channel, lpctstr filename, const time_t& from, const time_t& to, const bool* cameras, int length, bool usepassword, lpctstr password, bool includetextin = false, bool excludeplayer = false); the issue i'm having cameras argument; , i've had issue other const pointer arguments well. here's how i'm wrapping api: def search_exporttoclipcopy(self, hsearch, csearch, filename, tfrom, tto, cameras, length, usepassword, password, includetextin=true, excludeplayer=false): exporttoclipcopy = self.sdkdll.search_exporttoclipcopy exporttoclipcopy.argtypes = [c_int, c_int, c_wchar_p, c_long, c_long, pointer(c_bool), c_int, c_bool, c_wchar_p, c_bool, c_bool] exporttoclipcopy.restype = c_bool exporttoclipcopy(hsearch, csearch, filename, tfrom, tto, cameras, length, usepassword, pass
Read more