Converting video capture image to byte[]

ebkapalka

Hello! I'm trying to write an MJPEG streamer, and I'm having an issue. The stream should take a frame from the webcam and convert it to a byte[] array in the method extractBytes(PImage img) (which is currently empty). I've looked all over, and the best I can come up with is to convert the PImage into a BufferedImage and then convert that into a byte[] to send. My problem is that every example I can find is for extracting a single frame one time and saving it somewhere (instead of just grabbing it, converting it, and sending it without explicitly saving it anywhere in-between). A friend of mine who is pretty fluent in C# looked at the following code, and had the following remark: "I have no idea if it will function, but it's riddled with 'bad practices'". He proceeded to describe how the image sender should only send when a new image is available, and that the entire process should be done in a separate thread so that it can do the vision processing (oh yeah, this will eventually process Kinect images using Simple-OpenNI and OpenCV for Processing) simultaneously. I haven't used Processing in quite some time, so I had anticipated issues. Here's what I've got:

import java.io.BufferedReader;
import java.io.BufferedWriter;
import java.io.IOException;
import java.io.InputStreamReader;
import java.io.OutputStreamWriter;
import java.net.ServerSocket;
import java.net.Socket;
import java.net.UnknownHostException;
import processing.video.*;

PImage image;
Capture video;
Socket sock = null;
byte[] data = null;
OutputStream outputStream = null;

void setup(){
  video = new Capture(this,640,480);
  video.start();

  try   { 
    sock = new Socket("172.16.2.215", 4000);
    outputStream = sock.getOutputStream();   

    data = extractBytes(image);
    outputStream.write((
    "HTTP/1.0 200 OK\r\n" +
      "Server: YourServerName\r\n" +
      "Connection: close\r\n" +
      "Max-Age: 0\r\n" +
      "Expires: 0\r\n" +
      "Cache-Control: no-cache, private\r\n" + 
      "Pragma: no-cache\r\n" + 
      "Content-Type: multipart/x-mixed-replace; " +
      "boundary=--BoundaryString\r\n\r\n").getBytes());
  } catch (IOException c) { c.printStackTrace(); }
}

void draw(){
  if (video.available()) {
    video.read();
    image = video.get(0,0,video.width,video.height;  //supposedly returns a PImage
    data = extractBytes(image);
  }
  try   {
    outputStream.write((
      "--BoundaryString\r\n" +
        "Content-type: image/jpg\r\n" +
        "Content-Length: " +
        data.length +
        "\r\n\r\n").getBytes());
      outputStream.write(data);
      outputStream.write("\r\n\r\n".getBytes());
      outputStream.flush();
   } catch (IOException c) { c.printStackTrace(); }
}

byte[] extractBytes(PImage img){
  //need something here to convert PImage to byte[]...
  //it looks like I need to convert PImage to BufferedImage to byte[]
}

I apologize for my bad coding/question asking style. Does anyone have any ideas about what to do (about the video to byte[] or general functionality)? Thanks!

GoToLoop

Dunno much. But just remember that a PImage is represented as a int pixels[]. And it's 4 bytes (32 bits). ~:>
And there's an undocumented function called thread("") that invokes a function as a separate Thread!

dimkir

I had same problem:

how to convert PImage in memory into sequence of bytes in "jpg" or "png" format, without using save() or saveFrame(). The trick was to "mutate" a bit original PImage.java which has methods to convert PImage to byte stream. In order to do that, I have made a class PImage2.java which is based on PImage.java. And I have added some helper methods, which would allow to convert PImage to byte array:

https://bitbucket.org/dimkir/sketchshot/src/bc108706ee65f136bdab1a2cc48dd71ec25f06ff/src/org/sketchshot/utils/PImage2.java?at=master#cl-79

here's the link to the method: public ByteArrayInputStream getPImageAsInputStream(PImage img, String formatExtension);

You will have to do it in these steps:

a) add PImage2.java to your sketch

b) in your sketch instantiate PImage2 pimag2 = new PImage2(width,height,ARGB);

c) InputStream thisStreamWillContainRawJpgBytes =pimag2.getPImageAsInputStream(pImageToConvertToJpg, "jpg");

d) now you can read your stream thisStreamWillContainRawJpgBytes as regular stream (as if it were stream from opening file). You can for example use this method:

http://stackoverflow.com/a/1264737/1168382

To convert it to byte[] array.

ps. it is a bit "complicated" process, but that's the best I could find to get compressed PImage bytes data in memory.

ebkapalka

Hey! I forgot to follow up on this :P First of all, thank you all for your help! I ended up using bits of information from both of your responses to acieve my goal. Here it is:

void broadcast(PImage img) {
      BufferedImage bimg = new BufferedImage( img.width,img.height, 
       BufferedImage.TYPE_INT_RGB );    
      img.loadPixels();
      bimg.setRGB( 0, 0, img.width, img.height, img.pixels, 0, img.width);
      ByteArrayOutputStream baStream    = new ByteArrayOutputStream();
      BufferedOutputStream bos      = new BufferedOutputStream(baStream);
      try {
           ImageIO.write(bimg, "jpg", bos);
      } 
      catch (IOException e) {
            e.printStackTrace();
      }
      byte[] packet = baStream.toByteArray();
      //send this somehow
}

Sorry for the late response! Thanks!