We closed this forum 18 June 2010. It has served us well since 2005 as the ALPHA forum did before it from 2002 to 2005. New discussions are ongoing at the new URL http://forum.processing.org. You'll need to sign up and get a new user account. We're sorry about that inconvenience, but we think it's better in the long run. The content on this forum will remain online.
Page Index Toggle Pages: 1
Async String Loader (Read 2504 times)
Async String Loader
Apr 12th, 2010, 10:29am
 
Hi all, after a 6 months absence i have got back to Processing.  I was trying to build a requestStrings() loader using requestImage() as a basis.  Typically i am using it to make requests into an API.  In this case www.etsy.com database is what the ETSY API is interrogating and i am asking for the typical stuff like user data and relationships between users.  I make upwards of 1000-2000 call in maybe 30 secs, mixed in with requestImage() calls also.  There are some differences between requestStrings() and requestImage() in that requestStrings() has some error retrying code embedded and also everything around the actual string handling is different of course.  I have been using the code with 0184.

Couple of caveats...the code does work correctly under considerable real life load, however i have not yet build a unit test.  The layout is still a bit untidy and you can see my considerable abuse of println as a development tool ;o)

Ideas, thoughts, bugs, errors etc is what i am looking for.  And here is the code itself ..

volatile int requestsInPlay = 0;
     public int requestStringsMax = 10;
     volatile int requestStringsCount = 0;
     
     public class RequestStringsHolder{
           
           public String vessel[];
           int loaded;  //-2 we are in the process of loading, -1 not started to load yet, 0 failed load, 1 loaded
           
           public RequestStringsHolder() {
                 
                 loaded = -1;
           }
           
           public int getLoaded()
           {
                 return(loaded);
           }
     }
     
     
     public RequestStringsHolder requestStrings(String filename) {
           
           RequestStringsHolder holder = new RequestStringsHolder();
           requestsInPlay++;
           //println("Incoming requestsInPlay "+requestsInPlay+" " +filename+"");
           return requestStrings(filename, null, holder);
     }
     
     
     public RequestStringsHolder requestStrings(String filename, String extension, RequestStringsHolder holder) {
           AsyncStringsLoader ail = new AsyncStringsLoader(filename, extension,holder);
           ail.start();
           
           return holder;
     }
     
     
     class AsyncStringsLoader extends Thread {
           String filename;
           String extension;
           RequestStringsHolder holder;
           int retryLoad;
           
           public AsyncStringsLoader(String filename, String extension, RequestStringsHolder holder) {
                 this.filename = filename;
                 this.extension = extension;
                 this.holder = holder;
                 retryLoad = 3;      //something better than one but not having tons of exception messages spewing out
                 holder.loaded = -2; //say we are loading
                 
           }
           //this Runnable Interface basically makes this chunk of code run on a seperate thread as soon as this object is created
           public void run() {            
                 
                 //run this code
                 
                 
                 while(holder.vessel == null && retryLoad-- >0)
                 {
                       while (requestStringsCount == requestStringsMax) {      
                             
                             //println("o "+requestStringsCount+" "+requestStringsMax+"inplay "+requestsInPlay+" " +filename+"");
                             try {
                                   Thread.sleep(10);
                             } catch (InterruptedException e) { }
                       }
                       requestStringsCount++;
                       //println("BEFORE AsyncStringsLoader run("+(3-retryLoad)+")  " +System.nanoTime()+"");
                       
                       holder.vessel = loadStrings(filename);
                       try {
                             Thread.sleep(10);
                       } catch (InterruptedException e) { }

                 }
                 
                 //println("o "+requestStringsCount+" "+requestStringsMax+"inplay "+requestsInPlay+" " +filename+"");
                 //println("!!!!!!!!!!!!!!AFTER AsyncStringsLoader run()  " +System.nanoTime()+"");

                 // An error message should have already printed
                 if (holder.vessel == null) {
                       holder.loaded = 0;
                       requestsInPlay--;
                       //println("@@@@@@@@@@@@@@@ Outgoing WE DIDNT GET THAT requestsInPlay "+requestsInPlay+" " +filename+"VESSEL ==NULL");
                       
                 }
                  else if (holder.vessel.length == 0) {
                       holder.loaded = 0;
                       requestsInPlay--;
                       //println("@@@@@@@@@@@@@@@ Outgoing WE DIDNT GET THAT requestsInPlay "+requestsInPlay+" " +filename+" VESSEL LENGTH == 0");
                       
                 }else {
                       
                       holder.loaded = 1;
                       requestsInPlay--;
                       //println("Outgoing requestsInPlay "+requestsInPlay+" " +filename+"");
                 }
                 requestStringsCount--;
                 
                 
           }
     }
     
Re: Async String Loader
Reply #1 - Apr 13th, 2010, 2:44am
 
nikp wrote on Apr 12th, 2010, 10:29am:
I have been using the code with 0184.

Where did you found it Building from the source
Re: Async String Loader
Reply #2 - Apr 13th, 2010, 8:42pm
 
yes ...building from source.
Page Index Toggle Pages: 1