java - reading lots of messages from amazon sqs -


i trying read sqs queue bunch of messages on (~2mil) running in trouble how setup threading model decent performance.

i tried this:

            executorservice es = executors.newfixedthreadpool(16);             //getapproximatenumberofmessages() - uses sqs regular (not async) client call getqueueattributes, etc             final atomicinteger approximatenumberofmessages = new atomicinteger(getapproximatenumberofmessages());             while(approximatenumberofmessages.get() > 0) {                 es.execute(new runnable() {                     @override                     public void run() {                         //getmessages uses sqs regular (not async) client call receivemessage()                         list<message> messages = getmessages();                         (message message : messages) {                             handlesinglemessage(message);                         }                         approximatenumberofmessages.decrementandget();                     }                 });             } 

however, out of memory exceptions. tried switching threadpoolexecutor suggested here, when process seems lock up.

is there way process messages asynchronously without running out of memory?


Comments

Popular posts from this blog

javascript - RequestAnimationFrame not working when exiting fullscreen switching space on Safari -

Python ctypes access violation with const pointer arguments -