java - reading lots of messages from amazon sqs -
i trying read sqs queue bunch of messages on (~2mil) running in trouble how setup threading model decent performance.
i tried this:
executorservice es = executors.newfixedthreadpool(16); //getapproximatenumberofmessages() - uses sqs regular (not async) client call getqueueattributes, etc final atomicinteger approximatenumberofmessages = new atomicinteger(getapproximatenumberofmessages()); while(approximatenumberofmessages.get() > 0) { es.execute(new runnable() { @override public void run() { //getmessages uses sqs regular (not async) client call receivemessage() list<message> messages = getmessages(); (message message : messages) { handlesinglemessage(message); } approximatenumberofmessages.decrementandget(); } }); }
however, out of memory exceptions. tried switching threadpoolexecutor
suggested here, when process seems lock up.
is there way process messages asynchronously without running out of memory?
Comments
Post a Comment