c# - Read the large text files into chunks line by line -


suppose following lines in text file have read 

info  2014-03-31 00:26:57,829 332024549ms service1 startmethod - fillpropertycolor end info  2014-03-31 00:26:57,829 332024549ms service1 getreports_dataset - getreports_dataset started info  2014-03-31 00:26:57,829 332024549ms service1 cheduledgeneration - switchscheduledgeneration start info  2014-03-31 00:26:57,829 332024549ms service1 cheduledgeneration - switchscheduledgeneration limitid, subscriptionid, limitperiod, dtnextscheduleddate,shoplimittype0, 0, , 3/31/2014 12:26:57 am,0

i use filestream method read text file because text file size having size on 1 gb. have read files chunks in first run of program read 2 lines i.e. "getreports_dataset started of second line". in next run should read 3rd line. did code unable desired output.problem code doesn't give exact chunk have start read text in next run. , second problem while reading text lines .. don't give complete line..i.e. part missing in lines. following code:

readposition = getlastreadposition(); using (filestream fstream = new filestream(logfilepath, filemode.open, fileaccess.read, fileshare.readwrite)) using (system.io.streamreader rdr = new system.io.streamreader(fstream)) {     rdr.basestream.seek(readposition, seekorigin.begin);     while (numcharcount > 0)     {         int numchars = rdr.readblock(block, 0, block.length);          string blockstring = new string(block);         lines = blockstring.split(convert.tochar('\r'));         lines[0] = fragment + lines[0];         fragment = lines[lines.length - 1];          foreach (string line in lines)         {             lsttextlog.add(line);             if (lsttextlog.contains(fragment))             {                 lsttextlog.remove(fragment);             }             numprocessedchar++;         }         numcharcount--;     }     setlastposition(numprocessedchar, logfilepath); }

if want read file line-by-line, this:

foreach (string line in file.readlines("filename")) {     // process line here }

if must read line , save position, need save last line number read, rather stream position. example:

int lastlineread = getlastlineread(); string nextline = file.readlines("filename").skip(lastlineread).firstordefault(); if (nextline != null) {     lastlineread++;     setlastposition(lastlineread, logfilepath); }

the reason can't saving base stream position because streamreader reads large buffer full of data base stream, moves file pointer forward buffer size. streamreader satisfies read requests buffer until has read next buffer full. example, open streamreader , ask single character. assuming has buffer size of 4 kilobytes, streamreader this:

if (buffer empty) {     read buffer (4,096 bytes) base stream     buffer_position = 0; } char c = buffer[buffer_position]; buffer_position++;    // increment position next read return c;

now, if ask base stream's position, it's going report position @ 4096, though you've read 1 character streamreader.


Comments

Post a Comment

Popular posts from this blog

javascript - RequestAnimationFrame not working when exiting fullscreen switching space on Safari -

request animation frame stops working when exiting fullscreen clicking on original space's safari window. fine if fullscreen mode canceled escape key or calling cancelfullscreen(). steps reproduce: open https://dl.dropboxusercontent.com/u/769042/prezi/safari-fullscreen.html click "draw raf", kittie appears click "fullscreen", go fullscreen click "draw raf", kittie appears go space original safari was, showing "click exit fullscreen mode", click anywhere, out fullscreen click "draw raf", nothing happens what handling click calling: window.requestanimationframe(draw); which draws on canvas context: function draw() { ctx.drawimage(img, math.random()*500|0, math.random()*400|0, 100, 100); } i checked .hidden , .visibilitystate, updated correctly. tested on osx 10.9.3, safari 7.0.4 (9537.76.4). has workaround/solution other switching old settimeout? this sounds webkit bug 88940 : using reque...
Read more

jsf - How to ajax update an item in the footer of a PrimeFaces dataTable? -

this visual of table have: +---------+------------+-------------------+ | header1 | header2 | header3 | +---------+------------+-------------------+ | row | input | calc'ed output | | row b | input b | calc'ed output b | | etc.. | etc.. | etc.. | +---------+------------+-------------------+ | total: total calc'ed output | +------------------------------------------+ and stripped-down code: <p:datatable id="mytable" value="#{mybean.somelist}" var="currentitem"> <p:column headertext="header1"> <h:outputtext value="#{currentitem.name}" /> </p:column> <p:column headertext="header2"> <pe:inputnumber value="#{currentitem.inputval}"> <p:ajax event="change" listener="#{mybean.changelistener}" update="outputval outputtotal...
Read more

django - CSRF verification failed. Request aborted. CSRF cookie not set -

error: reason given failure: csrf cookie not set. in general, can occur when there genuine cross site request forgery, or when django's csrf mechanism has not been used correctly. post forms, need ensure: browser accepting cookies. view function uses requestcontext template, instead of context. in template, there {% csrf_token %} template tag inside each post form targets internal url. if not using csrfviewmiddleware, must use csrf_protect on views use csrf_token template tag, accept post data. you're seeing section of page because have debug = true in django settings file. change false, , initial error message displayed. can customize page using csrf_failure_view setting. did find, nothing helped, must foolish(( my template <form method="post">{% csrf_token %} {% field in form %} <div class="control-group"> {{ field }} </div> {% endfor %} <input type=...
Read more