› › ›

Multithreading Madness

colouredmirrorballcolouredmirrorball
edited June 2014 in Questions about Code

So I wanted to create an application that uses multithreading, to increase the performance. I encountered weird issues where threads started to influence each other, even though they shouldn't. Here is a sketch which reproduces the most important issue on my system:

// Simple sketch to illustrate how headache inducing multithreading can be.

//         WARNING     WARNING     WARNING
//      Do NOT  run this sketch if you are epileptic!
//               VERY seizure inducing!



ArrayList<Element> elements = new ArrayList<Element>();
DataProcess dataprocess = new DataProcess();  //In this object the multithreading happens.

void setup()
{
  size(600,600);
  smooth();

  for(int i = 0; i < 25; i++)
  {
    elements.add(new Element());
  }

  dataprocess.start();
}

void draw()
{
  background(0);

  //Draw the elements in Processing's main thread
  for(Element element : elements)
  {
    element.draw();
  }
}

void mousePressed()
{
  for(int i = elements.size()-1; i >=0; i--)
  {
    if(elements.get(i).checkMouseOver(mouseX, mouseY)) 
    {
      elements.get(i).startDragging(mouseX, mouseY);
      i = -1;
    }
  }
}

//Simple class demonstrating what I want to do. Elements can be moved around with the mouse.
class Element
{
  int sizex = 70;
  int sizey = 90;
  int x;
  int y;
  float r, g, b;
  color colour = color(0);

  int oldx, oldy;
  boolean dragging = false;

  Element()
  {
    x = (int) random(width-sizex);
    y = (int) random(height-sizey);
    r = random(1);
    g = random(1);
    b = random(1);
  }

  void draw()
  {
    fill(127);
    stroke(50);
    strokeWeight(3);
    rect(x, y, sizex, sizey, 7);

    noStroke();
    fill(colour);
    rect(x+5, y+5, sizex-10, sizey-10);
  }

  void update(int count)
  {
    if(dragging && !mousePressed) dragging = false;
    if(dragging)
    {
      x = mouseX - oldx;
      y = mouseY - oldy;
    }

    colour = color( (sin(r*TWO_PI+(float)count/200)*0.5+0.5)*255, (sin(g*TWO_PI+(float)count/200)*0.5+0.5)*255, (sin(b*TWO_PI+(float)count/200)*0.5+0.5)*255);
  }

  boolean checkMouseOver(int ix, int iy)
  {
    if(ix > x && ix < x+sizex && iy > y && iy < y+sizey) return true;
    else return false;
  }

  void startDragging(int ix, int iy)
  {
    oldx = ix-x;
    oldy = iy-y;
    dragging = true;
  }
}

class DataProcess extends Thread
{
  boolean running;
  int count;
  int prevTime = millis();
  float fr = 1000;    //Framerate of the separate thread, the higher, the more artifacts
  boolean limitedFramerate = true; //Disables the frame limiting, go as fast as it can!

  DataProcess()
  {
    running = false;
    count = 0;
  }

  void start()
  {
    running = true;
    super.start();
  }

  void run()
  {
    while (running)
    {
      boolean runIt = false;
      if(limitedFramerate)
      {
        if(millis() - prevTime > 1000.0/fr) runIt = true;
      }
      else runIt = true;
      if (runIt)
      {
        count++;
        try
        {

          for (Element element : elements)
          {
            element.update(count);
          }


        }
        catch(Exception e)
        {
          println(e);
        }
        prevTime = millis();
      }

    }
  }

  void quit()
  {
    running = false;
    interrupt();
  }
}

In this sketch, there are a number of Elements that can be moved around. They have a rectangle with a random colour, which changes in time. The colour is calculated in a separate thread.

If you set the boolean limitedFramerate at line 114 to false, the updating thread will go as fast as it can. On my system, this causes the whole sketch to psychedelicly flicker in all colours. What's interesting is that not only the rectangles change colour, but also the background and the stroke and colour of the surrounding rectangles! Why does this happen?

If you set limitedFramerate back to true and fr to a high enough number, you see the same effect where the updating thread affects things it shouldn't. Not bad enough as when the framerate of the updating thread is unlimited, but still bad enough to cause trouble. Even when the framerate is set to a pretty low number.

In the program I'm trying to create, these effects are pretty disturbing and actually a lot worse than in this simple example.

So... is this a bug in Processing? Does anybody know of a fix or workaround?

Best Answers

Answers

  • koogskoogs

    you can comment out line 32 and it still does it.

    line 91 seems to be the culprit, change it to be constant yellow, say, and the screen changes to yellow with occasional black flashes.

    can't explain why though - background isn't being called and nothing in the code is updating pixels[] explicitly

  • colouredmirrorballcolouredmirrorball

    Indeed. This is just a mock program, in the real program a lot of things happen, including changing colour variables. I'm trying to create some kind of synthesizer so multithreading sounded like it was great to reduce latency. Not sure why Processing messes it up so badly. I guess a memory address is called by both threads in a method somewhere?

  • KevinWorkmanKevinWorkman

    Something strange is definitely going on. I've simplified your code into an MCVE that demonstrates the problem:

    void setup() {
  size(200, 200);

  new MyThread().start();
}

void draw() {
  background(0);
}

class MyThread extends Thread {

  public void run() {
    while (true) {

      float r = random(255);
      float g = random(255);
      float b = random(255);

      color(r, g, b);

      try {
        //change this to a 1 for the problem to go away
        Thread.sleep(0);
      }
      catch(InterruptedException e) {
        e.printStackTrace();
      }

    }
  }
}

    Looking at this code, you should just see a black background. The thread doesn't even change anything, it just calls the color() method and throws the value away. Instead, the background flickers with random colors.

    Stranger still, if you replace the call to color with color(255, 255, 255), it STILL flickers random colors. I would at least expect it to display a white background.

    Calling Thread.sleep(1) seems to fix the problem though, which doesn't make a whole lot of sense either.

    I don't have any answers for you, but maybe this simplified example will get you closer to the actual problem.

  • KevinWorkmanKevinWorkman

    Doing a little more digging, both the color() function and the background() function call the colorCalc() function of PGraphics, which you can view here: https://github.com/processing/processing/blob/master/core/src/processing/core/PGraphics.java

    The colorCalc() method does have some variables that I wouldn't expect to be thread-safe, so I'm not really surprised stuff get screwed up when you use threads. The only part that surprises me is that it works when you call Thread.sleep(1).

    You could try synchronization, but IMHO that goes outside the scope that Processing was designed for.

  • PhiLhoPhiLho

    "You could try synchronization"
    Yes, I thought about this, but I am not a specialist of multithreading, so my feeble attempt wasn't successful; but interesting as it changed the kind of flickering...

    synchronized void draw()
    synchronized void update(int count)

    inside the class.

    KevinWorkman's hint is probably the best guess about the issue: Processing's internal are not designed for multithreading, the simple fact that fill() affects all further drawings is symptomatic. PApplet, PGraphics and friends have lot of shared state, plenty of opportunity to mess with parallel code.

    And I am not even sure that this usage of multithreading has a real gain (speed, reactivity) over the main drawing thread.

  • KevinWorkmanKevinWorkman

    For synchronization, you'd have to synchronize (on some lock) around anything accessing the same state, in this case the calls to background() and color(). Something like this:

    Object lock = new Object();

void setup() {
  size(200, 200);

  new MyThread().start();
}

void draw() {
  synchronized(lock) {
    background(0);
  }
}

class MyThread extends Thread {

  public void run() {
    while (true) {

      float r = random(255);
      float g = random(255);
      float b = random(255);

      synchronized(lock) {
        color(r, g, b);
      }


      try {
        //change this to a 1 for the problem to go away
        Thread.sleep(0);
      }
      catch(InterruptedException e) {
        e.printStackTrace();
      }
    }
  }
}

    This seems to fix the problem, but depending on your real use case, it negates any benefits that threading gives you. But like PhiLho said, I'm not sure this is a great use of threading in the first place.

  • colouredmirrorballcolouredmirrorball
    edited June 2014

    My intent is to do a lot of data processing (oscillating, real time input, sound wave generation, laser data processing) at the same time. If all of this gets executed after each other, things might slow down a lot. Decoupling drawing things on screen from the rest is just one step. The mock program in the first post is a very simplified example. Imagine the colours of the squares being determined by a lengthy calculation using real time inputs instead of a simple sine wave.

    So as it looks like, the solution would be to calculate colours in the graphic thread instead of the data processing threads? I took care to not use any display functions (fill, rect, ellipse etc.) in the data processing threads, so things like color() should be added to that list as well?

    I'm unfamiliar with synchronisation. Will look it up.

    Calling Thread.sleep(1) seems to fix the problem though, which doesn't make a whole lot of sense either.

    That actually makes a lot of sense, since the draw loop is very simple in the program, it probably only takes a couple nanoseconds and then does nothing for the rest of the 16.66666... ms before Processing calculates the next frame. If you use Thread.sleep(0), it updates continuously so the probability of it interfering with the draw loop is as good as 1. But if there is a ms interval time, the probability is severely reduced: it has to run in those couple nanoseconds the draw loop sets the background. If you let it run for long enough, you will notice the erratic behaviour.

  • GoToLoopGoToLoop
    edited June 2014

    All Processing sketches got 1 shared canvas, which is an instance of PGraphics stored in a field named g.
    If another Thread needs to use any PGraphics related API, it should instantiate its own
    rather than mess w/ the shared g canvas!

    In relation to color() function, due to Processing's pre-processor many bugs, can't be used w/ dot operator!
    In order to overcome that particular bug, I've placed class MyThread in its own ".java" file.

    Check it out my enhanced fix for @KevinWorkman's version:


    "Canvas Threading Bug.pde":

    /**
 * Canvas Threading Bug (v2.0)
 * by  KevinWorkman (2014/Jun)
 * mod GoToLoop
 *
 * forum.processing.org/two/discussion/5886/multithreading-madness
 */

void setup() {
  size(300, 200, JAVA2D);
  new MyThread(this).start();
}

void draw() {
  background(0);
}

void mousePressed() {
  frame.setTitle(((MyThread.useCanvas ^= true)
    ? "Canvas" : "PGraphics") + " Mode");
}


    "MyThread.java":

    import static processing.core.PConstants.JAVA2D;

import processing.core.PApplet;
import processing.core.PGraphics;

class MyThread extends Thread {
  static final int DELAY = 0;

  final PApplet p;
  final PGraphics pg;

  static boolean useCanvas = true;

  MyThread(PApplet pa) {
    p = pa;
    pg = pa.createGraphics(1, 1, JAVA2D);
  }

  public void run() {
    for (;;p.delay(DELAY))
      if (useCanvas)  p.color((int)  p.random(0xff000000));
      else            pg.color((int) p.random(0xff000000));
  }
}
  • KevinWorkmanKevinWorkman
    edited June 2014

    Another approach is to do all of your processing on another thread, but to only call the methods that aren't thread-safe from the draw() function. In other words, set r, g, and b in another thread, then access them in the draw() function. Something like this:

    float r = random(255);
float g = random(255);
float b = random(255);

void setup() {
  size(200, 200);

  new MyThread().start();
}

void draw() {
  background(0);

  color c = color(r, g, b);
  fill(c);
  rect(50, 50, 100, 100);
}

class MyThread extends Thread {

  public void run() {
    while (true) {

      r+= -5 + random(10);
      g+= -5 + random(10);
      b+= -5 + random(10);

      r = Math.max(0, Math.min(255, r));
      g = Math.max(0, Math.min(255, g));
      b = Math.max(0, Math.min(255, b));

      try {
        //change this to a 1 for the problem to go away
        Thread.sleep(0);
      }
      catch(InterruptedException e) {
        e.printStackTrace();
      }
    }
  }
}

    Note that you still might run into a case where the red is set to the next value, but the green and blue values haven't been processed yet. Not sure how big a deal that is, and if it is a big deal, you're back to needing synchronization.

    However, you mentioned that you're doing this because you might see a performance hit. This seems a bit like premature optimization to me, which as we all know is the root of all evil.

  • colouredmirrorballcolouredmirrorball

    So the question is, which methods are safe and which ones aren't?

    There are many more reasons to use multithreading. The goal is to do lots of things at once. The goal is to create a shape that can be projected with a laser, using real time inputs such as MIDI, OSC, Leap Motion gesture control, maybe webcam, ... Additional outputs such as sound and DMX are also planned. It's a node based program and the slogan is "connect everything to anything". Each of these elements have their own quirks and problems. If one part freezes up, I want the user to be able to remove/bypass it. Also, some parts will need to use their own framerate.

    For example, if something goes wrong in one part of the program, the part that deals with laser output does not get updated in a single threaded program. This means the data that is sent to the laser controller might stay in place if the controller is badly implemented (and some of them are). This results in the same frame being projected continuously, which does not only ruin the show but also imposes a real safety issue! Because if the program can lock up, it will do so just when something goes wrong and you need to disable the laser as soon as possible. (Of course regulations say that a laser should always be able to be shut off with an e-stop in hardware, but regulations are there to be ignored... )

  • GoToLoopGoToLoop
    edited June 2014 Answer ✓

    So the question is, which methods are safe and which ones aren't?

    The programmer who wrote a Java class should tell whether it's thread-safe or not!
    AFAIK, all Processing's API is mono-thread! Although there are some workarounds
    like instantiating our own PImage & PGraphics objects!
    Have you run my example above using such simple & easy technique? I-)

  • KevinWorkmanKevinWorkman
    Answer ✓

    Like GoToLoop said, anything that calls PGraphics or PApplet methods are probably not thread safe. And like I said, you can still do all of your calculations on another thread, just don't use the results of those calculations to call Processing functions until you're in the draw() function. You wouldn't have to change much of your current setup to take that approach.

    But are you sure you want to do all of this in Processing instead of Java?

  • KevinWorkmanKevinWorkman

    Here is your original example, with the calculations kept on the other threads, but the call to color() only inside the draw() function:

    ArrayList<Element> elements = new ArrayList<Element>();
DataProcess dataprocess = new DataProcess();  //In this object the multithreading happens.

void setup()
{
  size(600, 600);
  smooth();

  for (int i = 0; i < 25; i++)
  {
    elements.add(new Element());
  }

  dataprocess.start();
}

void draw()
{
  background(0);

  //Draw the elements in Processing's main thread
  for (Element element : elements)
  {
    element.draw();
  }
}

void mousePressed()
{
  for (int i = elements.size()-1; i >=0; i--)
  {
    if (elements.get(i).checkMouseOver(mouseX, mouseY)) 
    {
      elements.get(i).startDragging(mouseX, mouseY);
      i = -1;
    }
  }
}

//Simple class demonstrating what I want to do. Elements can be moved around with the mouse.
class Element
{
  int sizex = 70;
  int sizey = 90;
  int x;
  int y;
  float r, g, b;

  int oldx, oldy;
  boolean dragging = false;

  Element()
  {
    x = (int) random(width-sizex);
    y = (int) random(height-sizey);
    r = random(1);
    g = random(1);
    b = random(1);
  }

  void draw()
  {
    fill(127);
    stroke(50);
    strokeWeight(3);
    rect(x, y, sizex, sizey, 7);

    noStroke();
    color c = color(r, g, b);
    fill(c);
    rect(x+5, y+5, sizex-10, sizey-10);
  }

  void update(int count)
  {
    if (dragging && !mousePressed) dragging = false;
    if (dragging)
    {
      x = mouseX - oldx;
      y = mouseY - oldy;
    }

    r= (sin(r*TWO_PI+(float)count/200)*0.5+0.5)*255;
    g= (sin(g*TWO_PI+(float)count/200)*0.5+0.5)*255;
    b = (sin(b*TWO_PI+(float)count/200)*0.5+0.5)*255;
  }

  boolean checkMouseOver(int ix, int iy)
  {
    if (ix > x && ix < x+sizex && iy > y && iy < y+sizey) return true;
    else return false;
  }

  void startDragging(int ix, int iy)
  {
    oldx = ix-x;
    oldy = iy-y;
    dragging = true;
  }
}

class DataProcess extends Thread
{
  boolean running;
  int count;
  int prevTime = millis();
  float fr = 1000;    //Framerate of the separate thread, the higher, the more artifacts
  boolean limitedFramerate = false; //Disables the frame limiting, go as fast as it can!

  DataProcess()
  {
    running = false;
    count = 0;
  }

  void start()
  {
    running = true;
    super.start();
  }

  void run()
  {
    while (running)
    {
      boolean runIt = false;
      if (limitedFramerate)
      {
        if (millis() - prevTime > 1000.0/fr) runIt = true;
      } else runIt = true;
      if (runIt)
      {
        count++;
        try
        {

          for (Element element : elements)
          {
            element.update(count);
          }
        }
        catch(Exception e)
        {
          println(e);
        }
        prevTime = millis();
      }
    }
  }

  void quit()
  {
    running = false;
    interrupt();
  }
}

    Again, you might (and probably will) see situations where the r value is updated but the g and b values haven't. Not sure how big a deal that is to you.

  • GoToLoopGoToLoop
    edited June 2014

    IMO, the approach above is faulty!
    It could be eased a little by using 1 color variable instead of 3 separate r,g,b!

    What if there was more than 1 DataProcess instance.
    And all of them invoking Element's update() method at the same time?
    That would scramble all of those r,g,b fields for sure! >:)

    The saner approach is for each Thread to have its own PGraphics object!
    That way there'd be neither corruption states nor performance bottlenecks! *-:)

    The only problem would be to synchronize the time to render all those PGraphics to the main canvas! 8-|

  • KevinWorkmanKevinWorkman
    edited June 2014

    It could be eased a little by using 1 color variable instead of 3 separate r,g,b!

    I agree. However, the whole problem is that we can't use the color() function to create an instance of color. You could also roll it up into a single int or hexcode.

    What if there was more than 1 DataProcess instance. And all of them invoking Element's update() method at the same time?

    It is my understanding (assumption) that only one thread (DataProcess) will access any particular Element's update() function. Once we start talking about threading, there are all kinds of what-ifs we could run through, but I think OP's approach has only one Thread accessing any particular Element's update() function.

    The saner approach is for each Thread to have its own PGraphics object!

    I don't disagree. But again, the problem is that we can't use the PGraphics.color() function, apparently due to a bug in Processing's parser.

    The only problem would be to synchronize the time to render all those PGraphics to the main canvas!

    Would you really need to do that though? All we need the PGraphics for is to call the color() function. We don't actually have to draw to the PGraphics instance.

  • GoToLoopGoToLoop
    edited June 2014

    ... we can't use the color() function to create an instance of color.

    Function color() merely returns a primitive color value.

    Problem is that color() and many other PGraphics methods like background(), fill(), stroke(), etc,
    depends on protected colorCalc() method, which instead of returning the color result,
    spreads them in various fields like calcColor & calcAlpha, etc. That is 100% Thread unfriendly!

    ... we can't use the PGraphics.color() function, apparently due to a bug in Processing's parser.

    It's 100% a pre-processor bug! My very 1st example shows how to avert it by placing a class,
    which needs to call color() from another PGraphics, in a separate ".java" rather than a ".pde" file!

    We don't actually have to draw to the PGraphics instance.

    @colouredmirrorball gives me the impression that he's gonna push the envelope real hard soon! >:/

  • GoToLoopGoToLoop
    edited October 2020

    I've adapted 1 of the codes I've got here to use a double off-screen PGraphics in a separate thread("").

    The idea is while the separate Thread is working w/ 1 PGraphics,
    the main "Animation" Thread displays the other, and vice-versa.

    This is a very simple way to avoid multi-thread corruption methinks. Check it out:

    /**
 * Linked Balls (v4.4.0)
 * (Double Buffer Remix)
 *
 * by  NatashaGunaratna (2014/Feb)
 * mod Quark & GoToLoop
 *
 * Forum.Processing.org/two/discussion/5886/multithreading-madness#Item_17
 *
 * Forum.Processing.org/two/discussion/3087/
 * how-to-draw-lines-between-my-circles-where-they-overlap#Item_3
 *
 * Studio.ProcessingTogether.com/sp/pad/export/ro.989GaZC5t7EkE
 */

static final int SMOOTHING = 0;
static final int FPS = 60, DELAY = 1000/FPS >> 2;
static final int NUM = 0100, MIN = 020, MAX = 0100;

final Particle[] balls = new Particle[NUM];

static final String ENGINE = JAVA2D;
//static final String ENGINE = FX2D;
//static final String ENGINE = P2D;
//static final String ENGINE = P3D;

PGraphics mainCanvas, altCanvas;
boolean isMainCanvas, isLooping = true;

void setup() {
  size(800, 600, ENGINE);
  frameRate(FPS);

  //noSmooth();
  if (ENGINE != FX2D)  blendMode(REPLACE);

  for ( int i = 0; i != NUM; balls[i++] = new Particle(
    random(MAX>>1, width - MAX), random(MAX>>1, height>>1), 
    (int) random(MIN, MAX)) );

  mainCanvas = createCanvas(SMOOTHING);
  altCanvas  = createCanvas(SMOOTHING);

  thread("render");
}

void draw() {
  background(isMainCanvas? altCanvas : mainCanvas);
  //frame.setTitle("FPS: " + round(frameRate));
  surface.setTitle("FPS: " + round(frameRate));
}

void mousePressed() {
  if (isLooping ^= true)  loop();
  else                    noLoop();
}

void keyPressed() {
  mousePressed();
}

void render() {
  final PGraphics main = mainCanvas, alt = altCanvas;
  final Particle[] particles = balls;

  while (true) {
    final PGraphics pg = isMainCanvas? main : alt;
    final int fc = frameCount;

    pg.beginDraw();
    pg.background(-1);

    for (int j, i = 0; i < NUM; ) {
      pg.stroke(0);
      final Particle b = particles[j = i++].script(pg);

      pg.stroke(Particle.LINK);
      while (++j < NUM) {
        final Particle p = particles[j];
        if (b.isIntersecting(p))  b.linkToParticle(p, pg);
      }
    }

    pg.endDraw();
    isMainCanvas ^= true;

    if (fc == frameCount) {
      println();

      do {
        print(".");
        delay(DELAY);
      } while (fc == frameCount);
    }
  }
}

PGraphics createCanvas(final int smoothing) {
  final PGraphics pg = createGraphics(width, height, JAVA2D);

  pg.smooth(smoothing);

  pg.beginDraw();
  pg.colorMode(RGB);
  pg.ellipseMode(CENTER);
  pg.smooth();
  pg.noFill();
  pg.strokeWeight(2);
  pg.endDraw();

  return pg;
}

final class Particle {
  static final color LINK = #FF4040;
  static final float GRAV = .15;

  float x, y;
  final short d, r;
  float sx = random(.3, 1.5), sy = random(.1, .5);

  Particle(final float xx, final float yy, final int dd) {
    x = xx;
    y = yy;
    d = (short) dd;
    r = (short) (dd>>1);
  }

  Particle script(final PGraphics pg) {
    return move().bounce().gravity().display(pg);
  }

  Particle move() {
    x += sx;
    y += sy;

    return this;
  }

  Particle bounce() {
    if (x > width  - r | x < r) {
      sx *= -1.;
      move();
    }

    if (y > height - r | y < r) {
      sy *= -1.;
      move();
    }

    return this;
  }

  Particle gravity() {
    sy += GRAV;
    return this;
  }

  Particle display(final PGraphics pg) {
    pg.ellipse(x, y, d, d);
    return this;
  }

  Particle linkToParticle(final Particle p, final PGraphics pg) {
    pg.line(x, y, p.x, p.y);
    return this;
  }

  boolean isIntersecting(final Particle p) {
    return sq(p.x - x) + sq(p.y - y) < sq(p.r + r);
  }
}
  • colouredmirrorballcolouredmirrorball
    edited June 2014

    But are you sure you want to do all of this in Processing instead of Java?

    I'll abandon the Processing IDE for Eclipse somewhere soon. But I really like the graphical functions of Processing as well as the libraries.

    @GoToLoop, drawing on canvas is only done in Processing's draw thread, using a separate PGraphics instance is a bit overkill to only do things like colour calculation and such. But it is definately a solution for when you want to use multiple threads for drawing, so it's surely something that needs to be remembered!

    Thanks everyone for the input! Now off to push some boundaries... Looks like I'll need to go through my code and search for all color(), red(), green(), blue() etc. methods that will get executed in a separate thread from the main draw thread.

  • KevinWorkmanKevinWorkman

    You can still use Processing's visualization stuff in eclipse, even inside a Java application. You can embed a Processing applet inside Java code. After all, Processing is just Java!

    Drawing from multiple threads won't really work. All drawing is done on the EDT.

Sign In or Register to comment.