Spring XD Processing files -


we have requirement need process multiple files folder (each file having 50k-100k records) , after doing calculation store data in database. once content of file processed need trigger second stream further processing.

the way have designed solution is

file reader -- file splitter -- processor -- re-sequencer (release strategy per record) -- sink (load data in database)

the issue above design re-sequencer , sink not able figure out if contents of file stored in database without cannot invoke second stream.

we have done benchmark activities aggregator , seems slower re-sequencer. there known performance issues aggregator? there serious performance degradation if have release strategy of releasing when data re-sequencer

thanks, jayadeep

processor 1 - splits lines 

processor 2 - calculation on each message <int:service-activator input-channel="input" ref="singlepointcalculator" method="onmessage"/> <bean id="singlepointcalculator" class="com.processor.singlepointcalculator">                 <constructor-arg index="0" ref="output"/>  </bean>  sink  <int:resequencer input-channel='input' output-channel='reseqchannel' release-strategy-expression="size() == 1" ></int:resequencer>  <int:transformer ref="maptransformer" input-channel="reseqchannel" output-channel="mapchannel"/>  <bean id="maptransformer" class="com.sink.maptransformer"> <property name="columns" value=" side,value" /> </bean>  <int-jdbc:outbound-channel-adapter                 data-source="datasource"                 channel="mapchannel"                 query=" "> </int-jdbc:outbound-channel-adapter>   sink using aggregator  <int:aggregator input-channel='input' output-channel='aggchannel' " ></int:aggregator > <int:transformer ref="maptransformer" input-channel="aggchannel" output-channel="mapchannel"/>  <bean id="maptransformer" class="com.sink.maptransformer"> <property name="columns" value=" side,value" /> </bean>  <int-jdbc:outbound-channel-adapter                 data-source="datasource"                 channel="mapchannel"                 query=" "> </int-jdbc:outbound-channel-adapter>


Comments

Post a Comment

Popular posts from this blog

javascript - RequestAnimationFrame not working when exiting fullscreen switching space on Safari -

request animation frame stops working when exiting fullscreen clicking on original space's safari window. fine if fullscreen mode canceled escape key or calling cancelfullscreen(). steps reproduce: open https://dl.dropboxusercontent.com/u/769042/prezi/safari-fullscreen.html click "draw raf", kittie appears click "fullscreen", go fullscreen click "draw raf", kittie appears go space original safari was, showing "click exit fullscreen mode", click anywhere, out fullscreen click "draw raf", nothing happens what handling click calling: window.requestanimationframe(draw); which draws on canvas context: function draw() { ctx.drawimage(img, math.random()*500|0, math.random()*400|0, 100, 100); } i checked .hidden , .visibilitystate, updated correctly. tested on osx 10.9.3, safari 7.0.4 (9537.76.4). has workaround/solution other switching old settimeout? this sounds webkit bug 88940 : using reque
Read more

Python ctypes access violation with const pointer arguments -

i have api i'm trying wrap in python (2.7.6 on win7) code using ctypes. here's api: client_dllfunc bool clientapi search_exporttoclipcopy(clienthsearch handle, int channel, lpctstr filename, const time_t& from, const time_t& to, const bool* cameras, int length, bool usepassword, lpctstr password, bool includetextin = false, bool excludeplayer = false); the issue i'm having cameras argument; , i've had issue other const pointer arguments well. here's how i'm wrapping api: def search_exporttoclipcopy(self, hsearch, csearch, filename, tfrom, tto, cameras, length, usepassword, password, includetextin=true, excludeplayer=false): exporttoclipcopy = self.sdkdll.search_exporttoclipcopy exporttoclipcopy.argtypes = [c_int, c_int, c_wchar_p, c_long, c_long, pointer(c_bool), c_int, c_bool, c_wchar_p, c_bool, c_bool] exporttoclipcopy.restype = c_bool exporttoclipcopy(hsearch, csearch, filename, tfrom, tto, cameras, length, usepassword, pass
Read more