Adding additional options for code - reuse

mikechambers

From what I understand, when compiling, processing will include all PDEs in a directory into the final project / app. Could someone point to me where in the code the files are actually included? (in the preprocessor)? I searched through the preprocessing code, but could not find where the PDEs were included.

I am interested in adding an #include support directive so I can reuse code between projects. I believe this has been discussed and rejected in the past, but I am completely fine with just making the changes for my own use (I would also be happy to share them). (Alternative, I may just include a standard path to search for additional PDEs to include).

Can anyone point me in the right direction?

mike

amnon

I am seeing a lot of interesting things in the JavaBuild class, for example this.

Perhaps right after here could be a good place to add in any additional pde's? And then remove the original #include's before bigCode is sent through?

mikechambers

Thank you! That is exactly what I was looking for. There are two ways I could do this:

1. include directive. Probably the best from a code standpoint, as it allows you to only include the files you need. Might have to do some work to make sure code is not included multiple times.

2. Include folder. I think this would be the easiest to implement. Basically, specify a single folder of PDE files, which are always included in when compiling. This would probably be pretty easy to implement, but then you will often include code that is not used in a specific project.

I may hack together #2 really quickly and see how it works, and then look into doing #1.

Thoughts?

mike

amnon

Sounds like a good plan.

That way you can quickly assess if there are any major drawbacks to this route.

mikechambers

ok. I have this working.

You can grab the source and compile from my fork at: https://github.com/mikechambers/processing

Here is the info from the commit:

To enable, add the following preference to your preferences.txt file:

preproc.includes_path=/path/to/include/folder

When this preference is specified, any .pde or .java files included
in the includes folder will be available when compiling sketch files
from the IDE.

This allows you to place commonly used code and files in a single folder
and have them avaiable to all of your Processing projects, without
having to copy the files into each project directory.

If the preference is not specified, is empty, or does not point to
a folder, it will be ignored.

Note, I basically reused code for parsing SketchCode file in Sketch.java.

Ideally, this code should be refactored in a way that it can be reused (so we don't
have two copies of the code).

Ive tested on Mac and it works really well. Ill test on Windows later tonight.

If there is interest, Ill also post binaries later.

Note: This fork is based off of the latest code so it may be unstable or some things might not work.

PhiLho

Do you mean that all the code in the include folder will be part of all the sketches? Or what do you mean by "will be available"? Will there be a source-level mechanism to select which files to load?

mikechambers

This includes everything in the include folder. If you read through this thread, you will see there were two options discussed (including how difficult each would be to implement).

From above:

Include folder. I think this would be the easiest to implement. Basically, specify a single folder of PDE files, which are always included in when compiling. This would probably be pretty easy to implement, but then you will often include code that is not used in a specific project.

As noted above, the best solution is to add an #include directive for files. Since its not clear if this would ever make it into a release, I decided to see if I could get the easier option working (at least initially).

mike

mikechambers

Btw, If we did add support for #include, would we want it to:

1. Just directly include the included file into the main file?
2. Include the .pde or .java file to be parsed by the preprocessor?

You would not be able to include files in the same directory as your main file (at least not if they had the pde or java file extensions), and maybe not in some child directories that are used by the compiler (Im not 100% what those are, maybe "code" and "data").

amnon

Great! A good start is half the work. Now let's see if the solution can be refined as discussed above.

With regard to making it into the release, perhaps #include's would be an interesting feature for PDE X. This is an experimental mode currently being developed by Manindra Moharana with more advanced PDE features and it might perhaps someday become the default PDE of Processing (although simplicity for beginners will always be an essential value for Processing).

I think #included files should be parsed by the preprocessor (option 2), so that they may use Processing-specific syntax like any regular .pde. However, if these files are included into the main file (option 1) BEFORE the code is sent to the preprocessor, then the end result is likely to be the same, right?

You don't need to #include files in the same directory as your main file, because they are already automatically included by Processing. Processing adds .jar files that are in the code folder (aka libraries) and adds all files (regardless of the extension) that are in the data folder as input (aka text files, fonts, images). Indeed all of these locations are not suitable (should be disallowed) as source locations for #includes.

My $0.02. *-:)

mikechambers

Thanks for the info. I was looking at the code tonight, and I dont think it would be too difficult to implement:

1. Parse PDEs in the directory (code is already loaded into memory)
2. Look for include directive. If files are in same directory, ignore, otherwise, add specified path to list of files to be prepocessed.
3. Preprocess all files, include those specified in via includes.

The thing that could get tricky, is to make sure to only include any specific file once. I guess I could keep a list of files that have already been included, and then don't include once that is in that list already.

Also, what should the actual directive be? The standard is:

#include path

But I dont think that is valid processing / java code.

I would do something like:

//#include path

As that would still be valid processing code

Would also need to figure out if paths would be relative, or would we also support absolute (just relative would be easier).

mike

amnon

Indeed #include won't work in processing/java. Hence my original comment to remove the #include from the original file at the right time (between step 2 and 3). I think you can use the #include, after which the lines is removed by your custom code before it's sent to the preprocessor and thus long before it's compiled into valid java code. Of course, your solution would also work with the advantage that there is no extra work (no need to remove lines) but with the disadvantage that it's a comment. Your solution is easier to implement, so perhaps do that first? The proposed solution may then be the refinement?

My guess is that path support is trivial, so I would say both, if possible. Personally I think absolute paths are more suitable for specifying 'external code' since that way you are sure you can keep the same #include line, no matter where the sketch is located.

mikechambers

ok. I just about have this working. Here is how it will work (at least initially):

To include a file

#include RELATIVE_PATH_TO_FILE

The include file is relative to the main sketch file / pde. The include file will be ignored if it is in the same directory as the main pde.

Each file will only be included one, regardless of how many times you include it.

Initially, only files in the main file's folder will be scanned for include files, but I plan to add support for the include directive in included files.

If anyone is interested in help, one thing that would be useful is putting together, and testing a Regular Expression to search for include directives in a file (and get the include path):

Basically:

BEGINNING OF FILE(OPTIONAL WHITESPACE)"#include "(INCLUDE_PATH)EOL

Note, bonus points if it supports optional quotes (single and double) around the path. (I don't need them in my code, but some people will probably put them in).

Right now I look through each line of each file to find include directives, so having this regex should speed things up and / or simplify the code.

Anyways, if anyone wants to test the regex, it will help speed things up for me.

For the future, I may add support for absolute paths, and also search paths (similar to a Java Class path).

mike

mikechambers

Just a quick update, but I have this working (including the regex). Ill post an update later tonight (just cleaning a few things up, and looking for bugs).

I have tested on Windows and Mac, and it is working. If you want to try it out, grab the source from the link above and compile.

Usage:

#include RELATIVE_PATH_TO_INCLUDE_FILE

Example:

#include ../includes/Point.pde

Couple of notes:

1. Use a forward slash (/) to create relative paths.
2. Only looks for include files for files in the main sketch folder. i.e. Included files are not scanned for #include. (Ill plan to fix this).

mike

mikechambers

Known issues:

1. #include directives in include files are ignored

mike

amnon

Ah, some real progress with a working proof-of-concept.

If you want to spread this feature with others, you might want to think about what routes there are to do so. Because I doubt most people will download Processing's 1.7GB repo (bloated due to long development history) and compile themselves for just this feature. If they did, I suppose they needed the feature real bad! ;)

Some possibilities to get this feature out there:

1. Post as an issue first and possible pull later on the main processing repo. This would be the most ideal in terms of reaching the widest possible audience. Of course the devs are on point to what they do and don't want to include in the core to prevent bloat. One problem might be that #include is not something for beginners per se, which is always an important criterium. On other other hand, you never know. And even if not, perhaps you may get some suggestions from the devs with regard to other options (see following points).
2. Create a mode. Since I believe modes have control over compilation.
3. Integrate with an existing mode such as PDE X (as mentioned earlier).
4. Create a tool. Given the limitations of a tool (which I am not completely familiar with), this would most likely require different kind of solution. Possibly something like: indicate which files you want to include via a popup or default setting and then the tool copying those files into the folder before compiling.

Just sharing some ideas.

mikechambers

Yeah. Right now, I am just going to play with it. Ive been using it last night and today some, and for my workflow, it is awesome.

I may ping the dev team and see if they would be interested in it, and if so, how to move forward. Also, if there is interest, I might post binaries.

mike

mikechambers

ok. Just completely refactored this, and it is a lot more solid.

I need to use it a couple of days, but right now, the only known issue is:

1. #include directives in include files are ignored

You can grab the source and compile from my fork at: https://github.com/mikechambers/processing

If anyone is interested, I can post some binaries. Just let me know.

mike

amnon

Take a look at the release notes for the first Processing 3 alpha.

If there was ever a perfect timing to include includes... ;)

mikechambers

Yeah. When I get some time, I am going to add an issue for it on github and start a discussion to see if they would be interested (and if so, what would it take to make it happen).

Luckily, my changes are based off of a pretty recent build, and I feel my refactoring from this week is pretty solid architecturally. I still need to clean up the code though.

I have been using the build for the past couple of days, and it is working really, really well. I can't imagine using processing without it now.

mike

mikechambers

fyi, I submitted an issue in the bugbase to see if there is interest in including this into processing 3.0.

https://github.com/processing/processing/issues/2788

mike