Large Worlds with millions of objects

Schievelbein

I have been working on how to code my AI project, but keep running into memory limitations. Even with 64GB or RAM and a 4GB video card, I keep getting Out of Memory heap errors. I have modified my JVM settings and watch the resource usage. I am using all 63Gb or available memory. My PShape model when put into video memory only takes in a small percentage of the video RAM. The problem to the best of my knowledge is the creation of the hierarchical PShape World. The largest that I can get is about 40X40X40 brain cells, whereas I need 1K X 1K X 1K, but the memory requirements are virtually unobtainable in the near real world future.

Are there examples or Best Practices on using large worlds with many many objects?

As you can see from the image, there are 64000 cells in the object, but only ~1000 being viewed, how do I work on only those in a certain area or group?

Can all of the objects be stored in a database that is manipulated by Processing?

I have seen some stuff on unfolding, but do not understand it.

Any pointers to best practices or tutorials would be much appreciated.

clankill3r

What are you going to do with this? And do you always have this frontal view or is it possible to move around with a camera?

Schievelbein

Sorry, this is all within a PeasyCam world. I am able to rotate, zoom into the brain and select individual cells in order to view the cell characteristics. The different quadrants are have inputs to cells not shown on the diagram that are stimuli inputs from different senses. All of the cells have their rotation and colors controlled by stimuli and interaction with other cells. I have all of the specifics worked out, I just need to be able to scale this up to millions/billions of cells and be able to observe the trends and patterns on cells through different stimulation. The AI rules are quickly saturated on a 64000 cell brain. I would gladly post my project somewhere to share, but most of the sites will not work with the libraries...

clankill3r

Are you able to display the 1000x1000x1000 without the AI? (also, that's really a lot!) And just being able to see the code would help a lot (github?).

Schievelbein

Thank you very much for the suggestion of how to share my code. Could you please let me know if it works for you to download my code?

Yes, 1,000,000,000 cubes is the ULTIMATE goal I would be happy with 100x100x100 for the near future, but when designing the code, I wanted to design with the ultimate goal in case hardware was the limiting issue, as hardware is always changing and I have access to some really big cloud and cluster based systems if I could figure out how to design it correctly.

clankill3r

download link?

Schievelbein

DUHHH https://github.com/Schievelbein/Brain

clankill3r

I took a quit look. Hoping you did something stupid but i didn't find anything. Later i will take a better look. You could figure out a algorithm to guess what boxes probably can't be seen but that can be tricky.

I think i would move away to a proper 3d application with a good renderer. Redshift is really nice and fast and supports instancing: http://docs.redshift3d.com/Default.html#I/Instances.html

Redshift is also easy to use, i really loved it. Just get yourself a videocard with a lot of cuda cores.

You could also try Arnold, which relies not on your video card but the ram you have. And since you have 64gb :D solidangle.com Look at how they made gravity for example, i was amazed what was done by it. Even things i didn't expect.

Other options might be octane, fabric-engine or v-ray, but i don't know much about that. There is also another one but i forgot the name.

You still would need to set the render quality low to keep it to a low frameRate. Some renderers have some delay. I would check looping the timeline and see how that goes.

If your rich: That video is a bit out of context, it's not rendered by 1 video card but like 128 or something (can't remember how many, but a lot). This doesn't mean you have to buy that many cards, you pay to make use of the render cloud.

Then for a 3d application. Softimage is the best software i ever used (i tried 3dmax, blender, maya and cinema4d, some more then others). There is a reason the first 2 renderers i mentioned had very early support for for softimage (SI from now). SI supports JScript, python and VisualBasic. Then they have the sdk that supports c++ (and python as well if i'm correct). They also have node based programming (and the best implementation i have ever seen). The interface is really nice. The interaction with sliders etc. is the best i have seen. There is a construction history that allows you to change something you did earlier like smoothing etc. And 100 more things.

Here are 2 awesome videos:

God i love this product.

The only reason not to use it is because it's EOL cause AutoDesk is evil. To compare it to processing, imagine processing will discontinue and we all have to use actionscript or something...

This product allowed small companies or even one man companies to be a concurrent of larger companies.

If you consider to switch to a 3d application i recommend reading the letter below (not written by me).

Dear Mr. Bass

My name is Perry Harovas.

You don't know me, but I am a 10 year Softimage user. 10 years is actually a small amount of time when compared to my peers who having been using Softimage for up to 20 years.

I am writing to you because I cannot be silent on this.

I have been in this business for 25 years. I started out using Lightwave in Video Toaster V1 on an Amiga computer. I then moved on to Alias PowerAnimator and took the new abilities of that software (over Lightwave) into feature films out of a small studio in (of all places) Newark, NJ.

I was an Alpha tester of Maya, before it was even announced publicly. I put up with no docs, breaking code, a renderer that was written only months earlier and barely worked, changing workflows, etc. I learned everything I could about the software, and eventually co-authored the first book about Maya, "Mastering Maya Complete 2".

I was the loudest, most exuberant fan of Maya on the face of the planet. I couldn't get enough. I worked myself into bouts of sleeplessness in an effort to know more about this seemingly magical application that would allow me to create anything I could dream of.

Except, in reality, the word 'dream' is appropriate, because as I took on larger projects and tried to do more work with it, I found one of the largest obstacles with Maya was (and is) that it needs a support team behind it to code tools into either working together, or sometimes, working at all.

A good example of this is when I was directing two 30 minute CG children's shows with me and my small crew of 4 other people. We had 6 months to create 60 minutes of animation, including building the characters, rigging them, animating them, texturing, lighting, etc. An insane task given the budget, crew size and amount of animation. But we plunged head on into doing it.

Then, after many, many minutes of animation had been done, we found that our characters were coming into our scenes with no animation except their mouth lip sync. Where had all the animation we did gone?

Our one technical guy on staff looked into it and happened to find that the animation curves were still there, but had detached themselves from the character rig (his skeleton, if you will). Fortunately, he was able to code up a way to automatically reconnect the animation curves to the rig, saving months of work.

We then realized we were not going to be the only people to have this issue. We spoke with Support, and they acknowledged this was a known issue. We even offered to give them our script to help others who were having similar issues. They refused to let us help. We then started experiencing render problems, referencing issues, and a list of other things so long that I can't remember it now.

Needless to say, it was frustrating, it prevented the quality from being consistent, and endangered our whole company.

We soldiered on, finishing the two shows on schedule, barely, and vowing to NEVER use Maya again. We eventually decided on Softimage|XSI. Sure it was rough re-learning a new application, but it was rewarding in that it worked, didn't fail us, and didn't need a dedicated team to produce work that was better than what we could produce in Maya. This was astonishing to me! Thoughts of "Why did we not do this earlier?" ran through my head. The power in one application seemed to be nearly limitless.

Limitless, that is, until I started Alpha testing Moondust, which eventually became ICE. This was an area I knew nothing about, coding, and suddenly I was doing things that I could not believe. I created a way to have fur just appear on the silhouette of my cartoon dog, in literally 20 minutes of "fiddling around" with ICE.

Even with the lack of documentation at that point, with the alpha, and then beta, status of the software, it was the most powerful tool I had ever used.

Bar none. No doubt, No hyperbole.

I could not believe what I could now do, just ME, not a team of people. Imagine what a team of people could do? Well, there is no need to imagine, we have many examples to point to from just the last few years:

• 'The Lego Movie'
• The Mill's '98% Human' ad
• The Embassy's 'Science Project' commercial
• 'Iron Man'
• 'Pacific Rim'
• 'Now You See Me'
• Subaru 'Car Parts' ad

These are just off the top of my head.

This software, the one your company just retired (also known as EOL, or End Of Life) is Softimage. You remember Softimage, don't you? You bought it from Avid in 2008. I wouldn't blame you for not remembering, it never showed up on your home page, it was barely promoted, and it was something that you had to hunt for in Siggraph demos.

Softimage, the software that gave rise to dinosaurs in 'Jurassic Park' (in a previous, less powerful, incarnation of the software). Softimage, the software that gave the world 'Terminator 2'', 'Death Becomes Her', 'Babe, 'Casper', 'Jumanji', 'Mars Attacks' and just too many others to list. Softimage, the software that invented Inverse Kinematics. Softimage, the software that Animal Logic used to create the number one movie in America at the time Autodesk made this announcement.

Of course, I could go on. But I am sure you get the point.

This is the software that your company just killed. That really is the only word for it, KILLED. Killed, because it is now seen by many as obsolete even though, as you hopefully know, it is the newest of the 3 DCC apps your company owns and far from obsolete. It is now perceived that way because of this action.

I have been in visual effects and animation production my entire career, and a few years ago I also started teaching. In 2007, I moved myself and my family from the East Coast of the USA to Illinois. While there, I helped start a new college in Chicago called 'Flashpoint, The Academy of Digital Arts and Sciences', which was renamed to 'Tribeca | Flashpoint Academy' when Robert De Niro's Tribeca, purchased 50% of the school. I was the Chair of the Visual Effects and Animation Department. I wrote the entire curriculum in my department. The software world was 100% open to me, I could use anything I felt would be appropriate.

I chose to use Softimage, not Maya, as the main application to teach the students. I took on what I called an agnostic approach to teaching 3D software, in that students would learn the best tool for the job, and hopefully not be as software biased as my generation was/is. The reason for this, as I told them, was because you never know when your software will just vanish, the company will kill it, or go out of business.

Wow, I wish I wasn't right about that one...

Knowing how to use many applications, how to be aware of what was going on behind the curtain of the software, was (and is) far more important and helped prevent being unable to work due to not having skills in one application, especially if that application was discontinued.

Well, now because of the actions that your company took on March 4, 2014, that has happened, and I am in exactly that position. I can continue to use Softimage for a couple of years, but as you no doubt understand, the stigma associated with using EOL Software (never mind teaching it) is too great to bear. The driver support would quickly become a problem, the renderer support would be an major issue, and before too long, it will become impossible to use Softimage in production.

So your company has now given me, all of us Softimage users, a choice. We can use 3DS Max or we can use Maya. Well, I chose not to learn 3DS Max as it relied too heavily on plugins, and Maya, well, you now know how that played itself out for me...

I will make a choice, but it will be to not use Autodesk products anymore. I am choosing to learn an entirely new 3D DCC application, and I can assure you it will not be an Autodesk product. I could easily go back to using Maya, especially with my history with the software. I choose not to.

See? A choice.

Even if I thought Maya was a great base of code on which to build the future DCC application that will rule the world (and I certainly do not), I wouldn't use Maya based upon principal alone. Maya is not a great base to build upon, because as you are well aware, it is more than 17 years old and wasn't written when multi-core processors were even a dream in the labs of chip makers. It is really like tearing down the 5 story building, in favor of making a 10 story building on the foundation of a 17 year old house that has been patched together with one new idea after another. The foundation is showing its age, the wood is starting to rot, and yet the plan is to build on top of this base. That is what I believe you are doing with your company's plans to build upon Maya.

A company that does this to their loyal customers, in my opinion, is a company that does not have the best interests of those customers at heart. A company that does, what your company did in less than 6 years (killing a product) is a company that should not have made the purchase of that product in the first place.

Was it not obvious, when Autodesk evaluated the purchase of Softimage, that having three DCC apps was not a good financial decision? Was it not obvious that this fairly small market segment could not sustain development on these three apps?

If it wasn't obvious, then perhaps there is more wrong with Autodesk than anyone realized. Unless, there was another reason to buy Softimage, but that would be pure speculation...

So, what now?

I now teach at another well respected institution, and continue to work in the industry very actively. Will I still teach Maya? Yes. It is part of the curriculum, and will help these students get jobs in the industry.

However, I will be teaching another product as well, again, to prevent what has happened to me and my peers from happening to these students that are just starting out on this career path.

Mr. Bass, I have seen interviews with you. I believe you are a kind person. You certainly seem to be in love with this business. I hate the decisions that your company made, that you approved, but I don't personally hold any hatred for you.

I just wanted you to know that there are many, many artists out there that just want to continue to use the application that we chose to use. We had a choice, and Softimage was what we chose to use. Taking away that option, your company has now asked us to chose software that we could have chosen before, and did not. This isn't a choice, this is a hijacking.

I cannot speak for others, but, as you may now know, the overall sentiment of Softimage users is to chose NOT to use Autodesk products anymore. I can only believe this wasn't what Autodesk, or you, wanted. We are a very passionate bunch, who now are passionately opposed to using Autodesk products. Again, I can't speak for all of us, but the forums on your competitions' websites are filled with users looking to switch.

The most asked question on these forums seems to be something like: "How do I do this thing I used to do in Softimage, in your app?"

That should tell you something Mr Bass. The users of Softimage just want to keep using Softimage, its workflow, its amazing tools, and not be burdened by the archaic workflow of your other two DCC apps. They want to do amazing work. They want to be able to feed their families and to keep their employees.

They are so hungry for this, they are looking for this workflow in the welcoming arms of your competitors. The amount of money this may end up costing Autodesk in the not too distant future will most likely dwarf the amount of money it would have taken to just keep Softimage going.

I didn't go to business school, but even I can see this isn't a great way to run a business.

So, after this long email (that I actually edited believe it or not). where do I stand?

Well, I want to ask you to reconsider your company's decision. This cannot be an easy matter, and I am sure I don't know all the legal and financial problems this may create. But in a naive, passionate, and yes, tearful, way, I want to ask you:

Please bring back Softimage.

Please keep this software we all love going. Please just keep it alive in the most basic way. Keep fixing bugs, keep updating the support for graphics drivers, keep updating the SDK to allow the large community of developers to continue to enhance it.

Please do this for us, the people who put their faith, their financial dependence, in the software that your company owns.

Thank you for your time.

Sincerely,

Perry Harovas

clankill3r

I took a quick better look at your code (i'm really tired). First this is wrong:

CubeCelledBrain Brain;

Brain should be brain cause it's an instance. This won't matter for running but it matters for people who read your code.

Also following Java syntax rules (or whatever it's called) this is wrong: cells_per_side

Should be CellsPerSide

Then what you could try, create a PShape with only a certain amount of boxes instead of so many. For example 10x10x10. Then in draw, use pushMatrix, translate and popMatrix to go around and draw the 10x10x10 multiple times at multiple places.

This way it will take less memory. You should test with different numbers.

This goes really fast (7ms):

int cells_per_side = 1000;

    int start = millis();

    for (int i = 0; i < cells_per_side; i++) {
      for (int j = 0; j < cells_per_side; j++) {
        for (int k = 0; k < cells_per_side; k++) {
        }
      }
    }

    println(millis()-start);

But having a pushMatrix, translate and pop shows things really down (130519ms).

int cells_per_side = 1000;


void setup() {
  size(600, 600, OPENGL);
  int start = millis();

  for (int i = 0; i < cells_per_side; i++) {
    for (int j = 0; j < cells_per_side; j++) {
      for (int k = 0; k < cells_per_side; k++) {

        pushMatrix();
        translate(10, 10, 10);
        popMatrix();
      }
    }
  }

  println(millis()-start);
}

So bottom line is try to find the magic number, so a loop that doesn't take to long in combination with a PShape that holds a certain amount of cubes. You could also make a shape of the front. This way you only need one loop in the z direction (try that first i suggest).

Schievelbein

Thank you very very much for your help and guidance. I have made the changes you suggested and I rewrote the code to create small groups of cells and then cluster them into a larger [areas], and then cluster the ares into brain hemispheres and then cluster those into a brain. I think my issue with using some of the other suggestions, is that I need to be able to PICK any given cell and display the information stored within that cell. I think I am going to have to go the way of a stand alone renderer as you suggested and connect everything to a database to store the information. I believe this is more of a Data Visualization project and need to start thinking of it in more of that venue and see what I can figure out to do.

Thank you again for you help and I will sen you a link to the next version, as I appreciate your feedback.

clankill3r

yeah cool, keep me up to date! O yeah here is a good video of ice:

(If you use ICE you will need to create an ICE cache for the data).

For the rest, i think you should separate the visual and the data. In other words, don't store data in each cube but store it in a list (but you already mentioned database). Look into quadtree it might help: http://gamedevelopment.tutsplus.com/tutorials/quick-tip-use-quadtrees-to-detect-likely-collisions-in-2d-space--gamedev-374

I could imagine you only save data that is not default. Then in the quadtree you look at which parts are unchanged and render that with the 10x10x10 instance for example.

Anyway, you will run into problems, a project like this is not easy.