Speed of screenshots

mad_tunes

Hi Folks,

I'm brand new to processing having previously only messed around with Arduino, so please be gentle.

This may be a story you've read before... I'm toying with the idea of building an ambilight-like setup to run on my Server 2008r2/Kodi PC that plays through my TV. As much as a learning exercise as anything else. Processing>Arduino>Addressable RGB LEDs seems like a decent approach.

I thought I'd start with sampling the display in processing and work along the chain. After a bit of copy/pasting examples and googling I've got this as a very basic, very first step:

import java.awt.Robot;
import java.awt.*;
import java.awt.image.*;

boolean display = false;

PImage screenShot;

int thisTime = 0;
int lastTime = 0;
int frameTime = 0;

boolean prevMouseState = false;

void setup()
{
  size( displayWidth/2, displayHeight/2 );
  // background( 0 );
}

void draw()
{
  thisTime = millis();

  screenShot();

  frameTime=(thisTime-lastTime);
  lastTime=thisTime;

  if (mousePressed == true && mousePressed != prevMouseState) {
    display=!display;
  }

  if (display == true) {
    image( screenShot, 0, 0, width, height);
    fill(255);
    rect(10, 10, 55, 30);
    fill(0);
    textSize(26);
    text(1000/frameTime, 12, 35);
  }

  println(1000/frameTime);
}  

void screenShot()
{
  try {
    Robot robot = new Robot();
    screenShot = new PImage(robot.createScreenCapture(new Rectangle(0, 0, displayWidth, displayHeight)));
  } 
  catch ( AWTException e ) {
    e.printStackTrace();
  }
}  

I realise it's probably not the best test ever, but have I got something horribly wrong or could the laptop I'm trying this out on only be capable of grabbing the screen 7-10 times a second, and my media box only 45-50ish?

The laptop's a celeron with 4GB ram @1366*768, media PCs a core 2 duo with 8GB ram @1920x1080. Neither are monsters, but 7Hz with nothing else going on yet?!

Would 'grabbing' areas around the display be a better approach than grabbing it all at once to cut up later? (I haven't got this working yet so can't compare directly)

Like I said, I'm brand new to this, so please do state the obvious!

Thanks in advance

koogs

highlight code, press ctrl-o

mad_tunes

highlight code, press ctrl-o

that did it, thanks Koogs

GoToLoop

https://Processing.org/reference/frameRate.html

https://forum.Processing.org/two/discussion/8025/take-a-screen-shot-of-the-screen#Item_8

https://forum.Processing.org/two/discussions/tagged?Tag=createscreencapture()

mad_tunes

Switching to frameRate makes things look 3-4 better on the laptop.

Using your 'tweaked version w/ detailed library imports & cached pre-instantiations' with a println(round(frameRate)); added gives about 4!

I'm still reading...

GoToLoop

/**
 * Robot Screenshots (v3.15)
 * GoToLoop (2016-Mar-24)
 *
 * Forum.Processing.org/two/discussion/15674/speed-of-screenshots#Item_5
 * Forum.Processing.org/two/discussion/8025/take-a-screen-shot-of-the-screen
 */

import java.awt.Rectangle;
import java.awt.Robot;
import java.awt.AWTException;

PImage screenshot;

//static final String RENDERER = JAVA2D;
static final String RENDERER = FX2D;

static final int FPS = 60, DELAY = 1000/FPS/4, SMOOTH = 3;

void setup() {
  size(800, 600, RENDERER);

  smooth(SMOOTH);
  frameRate(FPS);
  imageMode(CORNER);

  screenshot = createImage(displayWidth, displayHeight, ARGB);
  thread("screenshotThread");
}

void draw() {
  image(screenshot, 0, 0, width, height);
  frame.setTitle("FPS: " + round(frameRate));
}

void screenshotThread() {
  final PImage shot = screenshot;
  final Rectangle dimension = new Rectangle(displayWidth, displayHeight);
  final Robot robot;

  try {
    robot = new Robot();
  }
  catch (AWTException cause) {
    exit();
    throw new RuntimeException(cause);
  }

  for (;; delay(DELAY))  grabScreenshot(shot, dimension, robot);
}

static final PImage grabScreenshot(PImage img, Rectangle dim, Robot bot) {
  bot.createScreenCapture(dim).getRGB(
    0, 0, 
    dim.width, dim.height, 
    img.pixels, 0, dim.width);

  img.updatePixels();
  return img;
}

mad_tunes

with static final String RENDERER = FX2D; i got: Cannot find anything named "FX2D"

switching it out for static final String RENDERER = JAVA2D; leads to FPS of 13-14 being reported.

Is repeatedly sampling the display just more 'work' than I realised?

GoToLoop

• FX2D is a Processing 3's exclusive renderer.
• I've got 60 FPS here in my laptop for both FX2D & JAVA2D renderers.
• But that FPS is the performance of displaying the PImage under the "Animation" Thread.
• The actual FPS of createScreenCapture(), which happens in a separate Thread, is unknown.
• You can control the delay between createScreenCapture() calls by changing constant DELAY.

mad_tunes

FX2D is a Processing 3's exclusive renderer.

That raises a question I glossed over when downloading the installer then... Processing v2 or v3? I know later is often greater, but I found more examples and general chatter about v2 so went with that. I assumed x64 on an x64 OS was the right move too, but I'll see how this runs on x86.

edit: x86 gives very similar numbers to x64

But that FPS is the performance of displaying the PImage under the "Animation" Thread. The actual FPS of createScreenCapture(), which happens in a separate Thread, is unknown.

I didnt realise they ran in seperate threads, I guess that means the frequency of grabbing the screen and displaying it might not be anywhere close to each other. They are both running from the same sketch though, one after the other in a loop, so should be 1:1. Which logic applies?

GoToLoop

Processing v2 or v3?

In most cases it doesn't matter. Only thing you'd "lose" is the new renderer FX2D if you choose P2.

I didn't realize they ran in separate threads, ...
They are both running from the same sketch though, ...

My posted code here is based on a very old 1 from: https://forum.Processing.org/two/discussion/8025/take-a-screen-shot-of-the-screen

The older 1 has both image() & createScreenCapture() being executed synchronously under the same "Animation" Thread. Which is the default sketch's Thread btW.

The newest 1 here relies on thread() in order to execute function screenshotThread() under another Thread concurrently: https://Processing.org/reference/thread_.html

mad_tunes

Ill try 3 out, its easy enough to switch.

I'm not sure I understand how multiple threads would work. Grab, chop, measure, pass on. Each needs the last to have finshed. Its all sequential isn't it?

GoToLoop

I'll try 3 out; it's easy enough to switch.'

Just make sure to choose a diff. path for its "sketchbook" folder, lest you corrupt your P2's! :-SS

It's all sequential isn't it?

Each Thread has its own "sequential" order. But they've got concurrent execution among each other.
For further specialized info about it, read these links:

1. https://Processing.org/reference/thread_.html
2. http://docs.Oracle.com/javase/8/docs/api/java/lang/Thread.html
3. https://docs.Oracle.com/javase/tutorial/essential/concurrency/index.html
4. http://tutorials.Jenkov.com/java-concurrency/index.html

mad_tunes

Using static final String RENDERER = FX2D; under 3.0.2 gives a good leap in the reported fps on my laptop. 45-50 reported on both my laptop and the media machine. (as long as that's pretty close to what I mean to be checking) That's a lot more like it.

Thank a lot

mad_tunes

I don't think frameRate is showing what I was hoping.

I'd set delay to 0 while trying to measure the screenshot frequency, I've just changed delay to 1000. The screenshot is changing every second, but the frameRate is still at 50+ times that.

So the two aren't as closely linked as I though I'd understood.

I think I'll add my own way to check how often it's happening back in, see what it shows.

GoToLoop

Variable frameRate is related to how many times draw() is invoked.
Therefore it gauges the "Animation" Thread only.

The other Thread executing screenshotThread() is of course completely separated from the "Animation" Thread.

The advantage of having the separate Thread dedicated for grabbing the screenshots is to free the "Animation" Thread from the obligation to await the whole process to complete before continuing.

For now, it mostly only displays the captured screenshot via image() and displays frameRate via setTitle() and nothing more.

Even though createScreenCapture() happens in a separate Thread, it doesn't make that process any faster. Rather it just eases up the "Animation" Thread.

In my example, constant DELAY is the milliseconds resting interval between each createScreenCapture().

Lesser the value, the faster another createScreenCapture() happens once previous 1 is finished.

Of course, just 1 millisecond or even 0 is very taxing to the CPU executing its Thread.

mad_tunes

Even though createScreenCapture() happens in a separate Thread, it doesn't make the process any faster. Rather it just eases up the "Animation" Thread.

Good way to put what I wasn't understanding. I think do now.

Of course, just 1 millisecond or even 0 is very taxing to the CPU executing its Thread.

Yup. This little old laptop feels like it might start to melt at any minute.