› › ›

JSON - how to get heirarchical names from file.

RichardDLRichardDL
edited March 2017 in How To...

I've just been given a 2.5MB JSON file example. The contents have about 5 levels of item, some with long lists of items. Ideally the function I want is: Return the heirarchial name a.b.c... of the item that has 'name' (input) as the value of a string called "text".

I've tried the JSON Load/save Array/Object functions in the help, but nothing leads me towards how to deal with a recursive tree structure. The examples contain the names of the structures, I'd prefer it read them from the file.

Tagged:

Best Answers

Answers

  • koogskoogs

    post what you've done and some test data (not the whole file, given the size). and format it nicely (highlight the code, press ctrl-o)

    generally it's expected that you know the shape of the data that you're receiving.

  • kfrajerkfrajer

    This could be relevant but not the final answer: http://crunchify.com/in-java-how-to-flatten-or-unflatten-complex-json-objects-into-flat-map-like-structure/

    You might need to do some reverse engineering and handle array cases and degenerate results (request values that has multiple entries in your json file).

    Kf

  • RichardDLRichardDL

    @koogs, what I've done is literally tried the help examples, and the data isn't mine to post. @kfrajer, spent some while looking at that, looked more involved than I expected. Then realised that as the given file is perfectly indented, what little I want is available by finding the item, then going back up the file watching the indents decrease. Thanks and sorry that wanting it now is making me use the quick crude way.

  • koogskoogs

    Test data, data like the data, it doesn't have to be THE Data, just the same shape. Without it everything is too theoretical.

  • koogskoogs

    the source is here: https://github.com/processing/processing/blob/master/core/src/processing/data/JSONObject.java

    annoyingly there is a map inside it, but it's private and therefore unreadable.

    i had a quick look and tried this, recursively parsing each element of the json and placing it in a map as i went, like you would a directory tree. BUT it doesn't work - parseJSONObject chokes on arrays, in a way you can't catch and handle. and because you don't know whether the object is an array or an object before you parse it then you are stuck. (this is what i mean about knowing the shape of the json, it would be trivial then. the general case is much harder.)

    // flatten json into a hashmap
// DOES NOT WORK

HashMap<String, Object> map = new HashMap<String, Object>();

void setup() {
  // json string (with 's instead of "s for ease of typing)
  // taken from the examples
  String input = "{"
    + "'bubbles': [{"
      + "'position': { 'x': 160, 'y': 103 }," 
      + "'diameter': 43.19838," 
      + "'label': 'Happy'"
    + "}, {" 
      + "'position': { 'x': 372, 'y': 137 }," 
      + "'diameter': 52.42526," 
      + "'label': 'Sad'"
    + "}]"
  + "}";

  // replace the 's with "s
  input = input.replaceAll("'", "\"");
  println(input);
  JSONObject jobj = parseJSONObject(input);
  flattenJson("", jobj);

  println("done");
}

// add elements to map
void flattenJson(String name, JSONObject json) {
  println(name);
  map.put(name, json);
  for (Object key : json.keys()) {
    println(key);
    String keyStr = (String)key;
    JSONObject next = json.getJSONObject(keyStr);
    println("Next", next);
    if (next != null) {
      // recurse
      flattenJson(name + "." + keyStr, next);
    } else {
      // maybe it's an array
      JSONArray array = json.getJSONArray(keyStr);
      if (array != null) {
        // TODO iterate over array
        println("Array: " + array);
      }
    }
  }
}

    will have a better look at it later, but i have things to do...

    (the source is a bit enlightening, they are parsing the json themselves rather than using one of the standard libraries to do it)

  • koogskoogs

    in a way you can't catch and handle

    this is false, it throws a runtimeexception. so that's good.

    the next problem is that get(int) for arrays is private... at least in the version of processing i'm currently using (was changed in oct 2016). downloading new version as i type...

  • koogskoogs
    Answer ✓

    this puts all the data into a hashmap, keyed by hierarchical name. haven't tested it beyond the data in the file. and 2.5MB might be too much for it but...

    // parsejson
// flatten json into a hashmap

import java.util.Map;

HashMap<String, Object> map = new HashMap<String, Object>();

void setup() {
  // json string (with 's instead of "s for ease of typing)
  String input = "{"
    + "'bubbles': [{"
    + "'position': { 'x': 160, 'y': 103 }," 
    + "'diameter': 43.19838," 
    + "'label': 'Happy'"
    + "}, {" 
    + "'position': { 'x': 372, 'y': 137 }," 
    + "'diameter': 52.42526," 
    + "'label': 'Sad'"
    + "}]"
    + "}";

  // replace the 's with "s
  input = input.replaceAll("'", "\"");
  //println(input);
  JSONObject jobj = parseJSONObject(input);
  flattenJson2(null, jobj);

  //println("done");
  println("Map:");
  for (Map.Entry e : map.entrySet()) {
    println(e.getKey(), e.getValue());
  }
}

void flattenJson2(String name, JSONObject json) {
  //println("name: " + name);
  for (Object key : json.keys()) {
    //println(key);
    String keyStr = (String)key;
    Object obj = json.get(keyStr);
    Class type = obj.getClass();
    //println("Object[" + keyStr + "]: " + type);
    if (type.equals(java.lang.String.class)
      || type.equals(java.lang.Integer.class)
      || type.equals(java.lang.Double.class)) {
      // this is a leaf so store
      map.put(toName(name, keyStr), obj);
    } else if (type.equals(processing.data.JSONObject.class)) {
      // it's a JSONObject
      JSONObject next = json.getJSONObject(keyStr);
      //println("Next", next);
      flattenJson2(toName(name, keyStr), next);
    } else if (type.equals(processing.data.JSONArray.class)) {
      // it's a JSONArray
      JSONArray array = json.getJSONArray(keyStr);
      for (int i = 0; i < array.size(); i++) {
        Object o = array.get(i);
        //println(i + ": " + o.getClass());
        flattenJson2(toName(name, keyStr) + "[" + i + "]", (JSONObject)array.get(i));
      }
    }
  }
}

String toName(String base, String key) {
  if (base == null) {
    return key;
  } else {
    return base + "." + key;
  }
}
  • RichardDLRichardDL 
    {
    "type": "picture",
    "displayName": "AFT",
    "asset": "",
    "version": "5.8.10383.15",
    "timeOfExport": "Monday, April 10, 2017 4:12:20 PM",
    "schemaVersion": "1.0.35363",
    "pictureDescription": {
        "name": "AFT",
        "left": 65535,
        "top": 65535,
        "width": 1000,
        "height": 608,
        "titlebar": false,
        "sysMenu": false,
        "bgColor": "#c0c0c0",
        "layer": -1,
        "shapeList": {
            "TWS_FOCUS": {
                "name": "TWS_FOCUS",
                "backDrop": {
                    "left": 59,
                    "top": -193,
                    "height": 167,
                    "width": 848,
                    "fill": "#868686",
                    "edgeWidth": 1,
                    "edgeColor": "#000000",
                    "edgeStyle": "solid",
                    "backgroundFill": "#ffffff"
                },
                "backDropVisible": false,
                "isSelectable": false,
                "type": "group",
                "left": 59,
                "top": -193,
                "height": 167,
                "width": 848,
                "bbLeft": 59,
                "bbTop": -193,
                "bbHeight": 166,
                "bbWidth": 848,
                "visible": true,
                "edgeWidth": 0,
                "edgeColor": "#000000",
                "edgeStyle": "none",
                "layer": -1,
                "fillColor": "#808080",
                "fillStyle": "solid",
                "backfillColor": "#ffffff",
                "shapeList": {
                    "TEXT": {
                        "name": "TEXT",
                        "type": "text",
                        "left": 189,
                        "top": -83,
                        "height": 56,
                        "width": 588,
                        "text": "(User configurable display)",
                        "fontFamily": "Arial",
                        "fontSize": 47,
                        "fontStyle": "bold",
                        "fontAlignment": "left",
                        "fontBaseline": "middle",
                        "edgeWidth": 0,
                        "edgeColor": "#000000",
                        "edgeStyle": "none",
                        "fillColor": "#808080",
                        "backfillColor": "none",
                        "visible": true,
                        "underline": false,
                        "strikethrough": false,
                        "maxLines": 1,
                        "maxCharsPerLine": 80,
                        "numLines": 1
                    },
                    "TEXT1": {
                        "name": "TEXT1",
                        "type": "text",
                        "left": 59,
                        "top": -193,
                        "height": 143,
                        "width": 848,
                        "text": "MAIN DISPLAY",
                        "fontFamily": "Arial",
                        "fontSize": 120,
                        "fontStyle": "bold",
                        "fontAlignment": "left",
                        "fontBaseline": "middle",
                        "edgeWidth": 0,
                        "edgeColor": "#000000",
                        "edgeStyle": "none",
                        "fillColor": "#808080",
                        "backfillColor": "none",
                        "visible": true,
                        "underline": false,
                        "strikethrough": false,
                        "maxLines": 1,
                        "maxCharsPerLine": 80,
                        "numLines": 1
                    }
                }
            },
            "TEXT2": {
                "name": "TEXT2",
                "type": "text",
                "left": 236,
                "top": 341,
                "height": 28,
                "width": 101,
                "text": "AF South",
                "fontFamily": "Arial",
                "fontSize": 24,
                "fontStyle": "normal",
                "fontAlignment": "left",
                "fontBaseline": "middle",
                "edgeWidth": 0,
                "edgeColor": "#000000",
                "edgeStyle": "none",
                "fillColor": "#000000",
                "backfillColor": "none",
                "visible": true,
                "underline": false,
                "strikethrough": false,
                "maxLines": 1,
                "maxCharsPerLine": 80,
                "numLines": 1
            },
            "TEXT3": {
                "name": "TEXT3",
                "type": "text",
                "left": 185,
                "top": 387,
                "height": 25,
                "width": 125,
                "text": "Nitrogen High",
                "fontFamily": "Arial",
                "fontSize": 22,
                "fontStyle": "normal",
                "fontAlignment": "left",
                "fontBaseline": "middle",
                "edgeWidth": 0,
                "edgeColor": "#000000",
                "edgeStyle": "none",
                "fillColor": "#000000",
                "backfillColor": "none",
                "visible": true,
                "underline": false,
                "strikethrough": false,
                "maxLines": 1,
                "maxCharsPerLine": 80,
                "numLines": 1
            },
            "TEXT4": {
                "name": "TEXT4",
                "type": "text",
                "left": 185,
                "top": 422,
                "height": 25,
                "width": 123,
                "text": "Nitrogen Low",
                "fontFamily": "Arial",
                "fontSize": 22,
                "fontStyle": "normal",
                "fontAlignment": "left",
                "fontBaseline": "middle",
                "edgeWidth": 0,
                "edgeColor": "#000000",
                "edgeStyle": "none",
                "fillColor": "#000000",
                "backfillColor": "none",
                "visible": true,
                "underline": false,
                "strikethrough": false,
                "maxLines": 1,
                "maxCharsPerLine": 80,
                "numLines": 1
            },
            "TEXT5": {
                "name": "TEXT5",
                "type": "text",
                "left": 185,
                "top": 456,
                "height": 25,
                "width": 186,
                "text": "Safety Keys Closed",
                "fontFamily": "Arial",
                "fontSize": 22,
                "fontStyle": "normal",
                "fontAlignment": "left",
                "fontBaseline": "middle",
                "edgeWidth": 0,
                "edgeColor": "#000000",
                "edgeStyle": "none",
                "fillColor": "#000000",
                "backfillColor": "none",
                "visible": true,
                "underline": false,
                "strikethrough": false,
                "maxLines": 1,
                "maxCharsPerLine": 80,
                "numLines": 1
            },
        },
        "animationList": {
            "AnVis": {
                "name": "AnVis",
                "type": "lookupAnimation",
                "expression": "AnVis_V1 == 0.00",
                "owner": "TWS_FOCUS",
                "animatedProperty": "visible",
                "exactMatch": true,
                "expressionTolerance": "0",
                "tolerance": "0.001000",
                "matchList": [
                    {
                        "loMatch": 1,
                        "outputType": "boolean",
                        "output": false,
                        "toggle": false
                    }
                ]
            },
            "AnFg": {
                "name": "AnFg",
                "type": "lookupAnimation",
                "expression": "AnFg_V1",
                "owner": "RECT",
                "animatedProperty": "foregroundColor",
                "exactMatch": false,
                "expressionTolerance": "0",
                "tolerance": "0.001000",
                "matchList": [
                    {
                        "loMatch": 0.0,
                        "hiMatch": 0.0,
                        "outputType": "color",
                        "output": "#ff0000",
                        "toggle": "#ff0000"
                    },
                    {
                        "loMatch": 0.0,
                        "hiMatch": 1.0,
                        "outputType": "color",
                        "output": "#00ff00",
                        "toggle": "#00ff00"
                    }
                ]
            },
            "AnFg1": {
                "name": "AnFg1",
                "type": "lookupAnimation",
                "expression": "AnFg1_V1",
                "owner": "RECT1",
                "animatedProperty": "foregroundColor",
                "exactMatch": false,
                "expressionTolerance": "0",
                "tolerance": "0.001000",
                "matchList": [
                    {
                        "loMatch": 0.0,
                        "hiMatch": 0.0,
                        "outputType": "color",
                        "output": "#ff0000",
                        "toggle": "#ff0000"
                    },
                    {
                        "loMatch": 0.0,
                        "hiMatch": 1.0,
                        "outputType": "color",
                        "output": "#00ff00",
                        "toggle": "#00ff00"
                    }
                ]
            },
            "AnFg2": {
                "name": "AnFg2",
                "type": "lookupAnimation",
                "expression": "AnFg2_V1",
                "owner": "RECT2",
                "animatedProperty": "foregroundColor",
                "exactMatch": false,
                "expressionTolerance": "0",
                "tolerance": "0.001000",
                "matchList": [
                    {
                        "loMatch": 0.0,
                        "hiMatch": 0.0,
                        "outputType": "color",
                        "output": "#ff0000",
                        "toggle": "#ff0000"
                    },
                    {
                        "loMatch": 0.0,
                        "hiMatch": 1.0,
                        "outputType": "color",
                        "output": "#00ff00",
                        "toggle": "#00ff00"
                    }
                ]
            },
            "AnFg3": {
                "name": "AnFg3",
                "type": "lookupAnimation",
                "expression": "AnFg3_V1",
                "owner": "RECT3",
                "animatedProperty": "foregroundColor",
                "exactMatch": false,
                "expressionTolerance": "0",
                "tolerance": "0.001000",
                "matchList": [
                    {
                        "loMatch": 0.0,
                        "hiMatch": 0.0,
                        "outputType": "color",
                        "output": "#ff0000",
                        "toggle": "#ff0000"
                    },
                    {
                        "loMatch": 0.0,
                        "hiMatch": 1.0,
                        "outputType": "color",
                        "output": "#00ff00",
                        "toggle": "#00ff00"
                    }
                ]
            },
        },
        "dataSourceList": {
            "AnVis_V1": {
                "tag": "#TWS_FOCUS"
            },
            "AnFg_V1": {
                "schemaVersion": "1",
                "systemType": "iFix",
                "systemName": "PMS",
                "bindingType": "RealTime",
                "nodeId": "PH908A1",
                "attributes": [
                    ".F_CV"
                ],
                "name": "AnFg_V1"
            },
            "AnFg1_V1": {
                "schemaVersion": "1",
                "systemType": "iFix",
                "systemName": "PMS",
                "bindingType": "RealTime",
                "nodeId": "PL908A1",
                "attributes": [
                    ".F_CV"
                ],
                "name": "AnFg1_V1"
            },
            "AnFg4_V1": {
                "schemaVersion": "1",
                "systemType": "iFix",
                "systemName": "PMS",
                "bindingType": "RealTime",
                "nodeId": "HS902ASJ",
                "attributes": [
                    ".F_CV"
                ],
                "name": "AnFg4_V1"
            },
            "AnVis1_V1": {
                "schemaVersion": "1",
                "systemType": "iFix",
                "systemName": "PMS",
                "bindingType": "RealTime",
                "nodeId": "XI902ASD",
                "attributes": [
                    ".F_CV"
                ],
                "name": "AnVis1_V1"
            },
            "AnFg5_V1": {
                "schemaVersion": "1",
                "systemType": "iFix",
                "systemName": "PMS",
                "bindingType": "RealTime",
                "nodeId": "XI902ASE",
                "attributes": [
                    ".F_CV"
                ],
                "name": "AnFg5_V1"
            },
            "AnVis2_V1": {
                "schemaVersion": "1",
                "systemType": "iFix",
                "systemName": "PMS",
                "bindingType": "RealTime",
                "nodeId": "XI902ASE",
                "attributes": [
                    ".F_CV"
                ],
                "name": "AnVis2_V1"
            },
            "AnVis3_V1": {
                "schemaVersion": "1",
                "systemType": "iFix",
                "systemName": "PMS",
                "bindingType": "RealTime",
                "nodeId": "XI902AND",
                "attributes": [
                    ".F_CV"
                ],
                "name": "AnVis3_V1"
            },
            "AnVis4_V1": {
                "schemaVersion": "1",
                "systemType": "iFix",
                "systemName": "PMS",
                "bindingType": "RealTime",
                "nodeId": "XI902ANE",
                "attributes": [
                    ".F_CV"
                ],
                "name": "AnVis4_V1"
            },
            "AnFg6_V1": {
                "schemaVersion": "1",
                "systemType": "iFix",
                "systemName": "PMS",
                "bindingType": "RealTime",
                "nodeId": "PH908A1",
                "attributes": [
                    ".F_CV"
                ],
                "name": "AnFg6_V1"
            },
            "AnFg7_V1": {
                "schemaVersion": "1",
                "systemType": "iFix",
                "systemName": "PMS",
                "bindingType": "RealTime",
                "nodeId": "PL908A1",
                "attributes": [
                    ".F_CV"
                ],
                "name": "AnFg7_V1"
            },
            "AnFg10_V1": {
                "schemaVersion": "1",
                "systemType": "iFix",
                "systemName": "PMS",
                "bindingType": "RealTime",
                "nodeId": "HS902ANJ",
                "attributes": [
                    ".F_CV"
                ],
                "name": "AnFg10_V1"
            }
        },
        "selectableGroupList": []
    }
}
  • RichardDLRichardDL
    Answer ✓

    @Koogs, Now I've had another look at the JSON parsing. I'm shocked to find it's over two months ago, anyway... Your program starts to work, then exceptions. I've made a small example file and posted, above.

    The exception is on line 59, ClassCastException: java.lang.String cannot be cast to processing.date.JSONObject.

    Taking the line apart: (JSONObject)array.get(i) does not like the text .F_CV .

    I've enabled your print statements and added some. Don't understand the order of processing, seems not in order down the input file. Difficult to tell if this is the first F_CV that it's arrived at.

    Just realised that I don't need the F_CV and if I chop them out of the source file (regex in N++) it runs through cleanly and gives a straight list of all the items, with hierarchical name - just what I was asking for in the first place :) wow thanks. I don't need the F_CV and could remove it on the fly, but a purist would want to understand and fix.

    The map output at the end is in a strange order, not the same as the input, not alpha. It probably doesn't matter but I think I want it alpha at each level. I'll look at the listing function. (And I've read+used HashMap for the first time - good.)

    Thanks again, Richard.

  • koogskoogs

    Will have a look later when I'm awake. But, yes, maps aren't stored in any order, they are meant to allow O(1) lookup, not to be printed as a list. It's easy enough to order them alphabetically by key though.

    (Actually a TreeMap should preserve the order and is a more obvious choice given the input is a tree)

  • koogskoogs 
                "attributes": [
                ".F_CV"
            ],

    these are the only arrays that don't contain json objects, which is why it's falling over.

    anyway, fixed (alphabetical also):

    // parsejson2
// flatten json into a hashmap

import java.util.Map;
import java.util.TreeMap;

TreeMap<String, Object> map = new TreeMap<String, Object>();

void setup() {
  // json string (with 's instead of "s for ease of typing)
  String input = "{"
    + "'bubbles': [{"
    + "'position': { 'x': 160, 'y': 103 },"
    + "'diameter': 43.19838,"
    + "'label': 'Happy',"
    + "'attributes': ["
    + "'attr1',"
    + "'attr2'"
    + "]"
    + "}, {"
    + "'position': { 'x': 372, 'y': 137 },"
    + "'diameter': 52.42526,"
    + "'label': 'Sad'"
    + "}]"
    + "}";

  // replace the 's with "s
  input = input.replaceAll("'", "\"");

  // load from file
  input = join(loadStrings("data.json"), "");

  //println(input);
  JSONObject jobj = parseJSONObject(input);
  handleJsonObject(null, jobj);

  //println("done");
  println("\n\nMap:");
  for (Map.Entry e : map.entrySet()) {
    println(e.getKey(), "=", e.getValue());
  }
}

void handleObject(String name, Object object) {
  Class type = object.getClass();
  //println("Name: ", name, " Type: ", type);
  if (type.equals(java.lang.String.class)
    || type.equals(java.lang.Integer.class)
    || type.equals(java.lang.Double.class)) {
    // this is a leaf so store
    map.put(name, object);
  } else if (type.equals(processing.data.JSONObject.class)) {
    // it's a JSONObject
    handleJsonObject(name, (JSONObject)object);
  } else if (type.equals(processing.data.JSONArray.class)) {
    // it's a JSONArray, process each element in turn
    JSONArray array = (JSONArray)object;
    for (int i = 0; i < array.size(); i++) {
      Object o = array.get(i);
      //println(i + ": " + o.getClass());
      handleObject(name + "[" + i + "]", o);
    }
  }
}

void handleJsonObject(String name, JSONObject jsonObject) {
  // loop over all the sub-objects in this object
  for (Object key : jsonObject.keys()) {
    //println(key);
    String keyStr = (String)key;
    Object next = jsonObject.get(keyStr);
    handleObject(toName(name, keyStr), next);
  }
}

String toName(String base, String key) {
  if (base == null) {
    return key;
  } else {
    return base + "." + key;
  }
}

    it expects the json to be in a file data.json in the usual data directory under the sketch.

  • koogskoogs 
    "selectableGroupList": []

    oh, empty arrays are omitted...

  • kasperkampermankasperkamperman

    Koogs thanks for the code and the post, very useful. I'm creating interface to the Microsoft Azure Cognitive Services Face API raw working code on Gist.

    I'm wondering if and how I can iterate over certain base keys?

    My output is now like:

    faceAttributes.age = 38.3 faceAttributes.emotion.anger = 0.0 faceAttributes.emotion.contempt = 0.001 faceAttributes.emotion.disgust = 0.0 faceAttributes.emotion.fear = 0.0 faceAttributes.emotion.happiness = 0.0 faceAttributes.emotion.neutral = 0.888 faceAttributes.emotion.sadness = 0.11 faceAttributes.emotion.surprise = 0.0 faceAttributes.facialHair.beard = 0.4 faceAttributes.facialHair.moustache = 0.5 faceAttributes.facialHair.sideburns = 0.5 faceAttributes.gender = male faceAttributes.glasses = NoGlasses faceAttributes.hair.bald = 0.02 faceAttributes.hair.hairColor[0].color = brown faceAttributes.hair.hairColor[0].confidence = 1.0 ... etc.

    I'd like to iterate over all the emotions and display the values. How would I do that? Some idea's or pointers to documentation would be welcome.

  • koogskoogs

    pointers to documentation

    https://forum.processing.org/two/discussion/21503/json-how-to-get-heirarchical-names-from-file

    (Sorry!)

    Your question is more about Java maps than json, but a simple string compare around line 40 should do it.

  • koogskoogs

    (line 40 of my code above, that is)

  • kasperkampermankasperkamperman

    Clear. I thought maybe something is build in Java treemaps, but I'll work with compare then.

  • koogskoogs

    the other way to do this is to pick out the emotion json object before calling my code:

    input = join(loadStrings("data.json"), "");
JSONObject jobj = parseJSONObject(input); // root
JSONObject faceAttributes = jobj.getJSONObject("faceAttributes");
JSONObject emotions = faceAttributes.getJSONObject("emotion");
handleJsonObject(null, emotions); // my method

    so you actually start parsing two levels into the structure. this requires you to know the structure of the data.

Sign In or Register to comment.