performance - Running time / Memory issue while copying from excel to SQL tables using Talend -


i copying data excel sheet sql tables. @ time around 2000 rows distributed across 18 tables. problem job is taking of time. takes around 2.5 mins job.

other issue facing memory. tried copy around 250,000 rows , couldn't run job basic settings. have increase xms , xmx allocation.

here ,y job design

how solve these issues?

you should start job tmssqlconnection (i think that's dbms you're using) , finish tmssqlcommit component , see if helps @ talend opening large amount of connections database rather pooling them.

increasing commit size speed bulk loads if fails commit lose entire commit.

as this, long have no race conditions , don't care in order tables inserted or updated parallelise whole job either tparallelize component or enabling multi thread executions in tab under job window.

sometimes memory usage in job can improved splitting process down separate jobs , linking them child jobs in 1 large wrapper parent job trunjob components. make job more manageable.

finally, there's couple of options in advanced settings of each database output component allows increase batch size (although increase memory usage) , enable parallel connections can improve performance utilising more database server cores.

your memory issues unlikely resolved short of re-engineering job deal smaller chunks of data @ time , commit each part , grab next lot.

this done using tfilterrow component , selecting first x records (by filter condition, if data set has none add 1 first preprocessing give every row numeric.sequence), processing , putting in table , picking next x records , on.


Comments

Post a Comment

Popular posts from this blog

javascript - RequestAnimationFrame not working when exiting fullscreen switching space on Safari -

request animation frame stops working when exiting fullscreen clicking on original space's safari window. fine if fullscreen mode canceled escape key or calling cancelfullscreen(). steps reproduce: open https://dl.dropboxusercontent.com/u/769042/prezi/safari-fullscreen.html click "draw raf", kittie appears click "fullscreen", go fullscreen click "draw raf", kittie appears go space original safari was, showing "click exit fullscreen mode", click anywhere, out fullscreen click "draw raf", nothing happens what handling click calling: window.requestanimationframe(draw); which draws on canvas context: function draw() { ctx.drawimage(img, math.random()*500|0, math.random()*400|0, 100, 100); } i checked .hidden , .visibilitystate, updated correctly. tested on osx 10.9.3, safari 7.0.4 (9537.76.4). has workaround/solution other switching old settimeout? this sounds webkit bug 88940 : using reque
Read more

Python ctypes access violation with const pointer arguments -

i have api i'm trying wrap in python (2.7.6 on win7) code using ctypes. here's api: client_dllfunc bool clientapi search_exporttoclipcopy(clienthsearch handle, int channel, lpctstr filename, const time_t& from, const time_t& to, const bool* cameras, int length, bool usepassword, lpctstr password, bool includetextin = false, bool excludeplayer = false); the issue i'm having cameras argument; , i've had issue other const pointer arguments well. here's how i'm wrapping api: def search_exporttoclipcopy(self, hsearch, csearch, filename, tfrom, tto, cameras, length, usepassword, password, includetextin=true, excludeplayer=false): exporttoclipcopy = self.sdkdll.search_exporttoclipcopy exporttoclipcopy.argtypes = [c_int, c_int, c_wchar_p, c_long, c_long, pointer(c_bool), c_int, c_bool, c_wchar_p, c_bool, c_bool] exporttoclipcopy.restype = c_bool exporttoclipcopy(hsearch, csearch, filename, tfrom, tto, cameras, length, usepassword, pass
Read more