Unfolding maps, tilemill and streaming tiles: howto

Share your Work

aart.rost

Unfolding maps, tilemill and streaming tiles: howto

in Share your Work • 2 years ago

Hi guys,

I'm doing a project where I need an detailed worldmap and stumbled my way into the amazing unfolding maps library. I created a simple addon so you can directly stream tiles from TileMill to processing. I was looking for something like this for ages and I'd like to share it with you :).

Step 0: If you are lazy: a sample sketch will be at the end of my post :).

Step 1: Download the Unfolding Maps library, GlGraphics library and install TileMill. You can choose to install TileMill locally on your mac or on a powerful Ubuntu 10.10/11.04 server if available.

Step 2: Import the following classes to your sketch:

import processing.opengl.*;
import codeanticode.glgraphics.*;
import de.fhpotsdam.unfolding.*;
import de.fhpotsdam.unfolding.core.*;
import de.fhpotsdam.unfolding.geo.*;
import de.fhpotsdam.unfolding.utils.*;
import de.fhpotsdam.unfolding.providers.*;

Copy my class to add support for tile streaming:

class streamProvider extends MapBox.MapBoxProvider {
String hostname, mapname;
streamProvider(String _hostname, String _mapname){
hostname = _hostname;
mapname = _mapname;
}
public String[] getTileUrls(Coordinate coordinate) {
String url = "http://"+hostname+":8889/1.0.0/"+mapname+"/"
+ getZoomString(coordinate) + ".png";
return new String[] {
url
};
}
}

And initialise your map:

map = new de.fhpotsdam.unfolding.Map(this, new streamProvider("localhost", "control-room"));

The class takes 2 variables. The host of the TileMill server and the name of the map it should stream. You can find the map name in the URL after clicking a project.

Step 3: Profit! You can edit (and don't forget to save) the map style and it will be directly streamed to your processing sketch. Just zoom in and out a bit to get rid of the cached tiles :).

Step 4, the lazy bastard step: here's a demo. It has some US cities in it to fill the screen a bit. Fire up TileMill and you should be set.

Enjoy and an enormous thanks to the creators of Unfolding Maps & TileMill!

Replies(22)

fishandch..

Re: Unfolding maps, tilemill and streaming tiles: howto

1 year ago

thanks a lot!

it kind of works but for some reasons i have problems loading the map as my local host for tilemill is 20009

and the root is http://localhost:20009/#/project/control-room

it seems like it doesn't upload the map.

How can i fix it?

i tried to change the host number in the stream provider class, but it doesnt work.

aart.rost

Re: Unfolding maps, tilemill and streaming tiles: howto

1 year ago

Hi, no problem :)

I think you use the new version of tilemill. They changed some stuff including the tile arrangement. Try to replace the old StramProvider class with this:

New StreamProvider:

class streamProvider extends MapBox.MapBoxProvider {
String hostname, mapname;
streamProvider(String _hostname, String _mapname) {
hostname = _hostname;
mapname = _mapname;
}
public String getZoomString(Coordinate coordinate) {
return (int) coordinate.zoom + "/" + (int) coordinate.column + "/" + (int) coordinate.row;
}
public String[] getTileUrls(Coordinate coordinate) {
String url = "http://"+hostname+": 20009/tile/"+mapname+"/"
+ getZoomString(coordinate) + ".png?updated=" + int(random(0, 10000));
return new String[] {
url
};
}
}

Let me know if that works out! (Don't forget to change the port bytheway)

______________________________________________________________

And something new. If you want to stream from a generated .mbtiles file:

class customMBTilesMapProvider extends AbstractMapTileProvider {
protected String jdbcConnectionString;
public customMBTilesMapProvider() {
super(new MercatorProjection(26, new Transformation(1.068070779e7f, 0.0f, 3.355443185e7f, 0.0f,
-1.068070890e7f, 3.355443057e7f)));
}
public customMBTilesMapProvider(String jdbcConnectionString) {
this();
this.jdbcConnectionString = jdbcConnectionString;
}
public int tileWidth() {
return 256;
}
public int tileHeight() {
return 256;
}
public PImage getTile(Coordinate coord) {
float gridSize = PApplet.pow(2, coord.zoom);
float negativeRow = gridSize - coord.row - 1;
return MBTilesLoaderUtils.getMBTile((int) coord.column, (int) negativeRow, (int) coord.zoom, jdbcConnectionString);
}
}

Instantiate like this:

map = new de.fhpotsdam.unfolding.Map(this, new customMBTilesMapProvider("jdbc:sqlite:" + dataPath("tiles/control-room.mbtiles") + ""));

fishandch..

Re: Unfolding maps, tilemill and streaming tiles: howto

1 year ago

Thanks,

yeas i'm using tilemill 0.9.0.

i tried and replaced my port number, but it's stil says:

"The file http://localhost:20009/#/project/tile/control-room/3/0/0.png?updated=3851 contains bad image data, or may not be an image."

here it is the class...

class streamProvider extends MapBox.MapBoxProvider {
String hostname, mapname;
streamProvider(String _hostname, String _mapname) {
hostname = _hostname;
mapname = _mapname;
}
public String getZoomString(Coordinate coordinate) {
return (int) coordinate.zoom + "/" + (int) coordinate.column + "/" + (int) coordinate.row;
}
public String[] getTileUrls(Coordinate coordinate) {
String url = "http://"+hostname+":20009/tile/"+mapname+"/"
+ getZoomString(coordinate) + ".png?updated=" + int(random(0, 10000));
return new String[] {
url
};
}
}

i'll try with a earlier version and thanks for the MBTiles version!!!

aart.rost

Re: Unfolding maps, tilemill and streaming tiles: howto

1 year ago

Hi,

I downloaded the latest version. This works for me:

1: change port to 20008, not 20009.

2: Instantiate like this:

map = new de.fhpotsdam.unfolding.Map(this, new newStreamProvider("localhost", "control-room"));

That should do it. Really annoying that those guys change everything with every release!

Good luck :)

fishandch..

Re: Unfolding maps, tilemill and streaming tiles: howto

1 year ago

It does!

Thanks a lot..saved my week!

aart.rost

Re: Re: Unfolding maps, tilemill and streaming tiles: howto

1 year ago

Good to hear! :).

Oh and by the way: If you want to use the mbtiles solution: Don't forget to include a sqlite driver. You can download a proper one on this page: http://www.zentus.com/sqlitejdbc/ . Simply put the .jar into a folder named "code" in the sketch root folder and you're set.

sirano_si..

Re: Unfolding maps, tilemill and streaming tiles: howto

1 year ago

Massive thanks for this aart.rost

If I can just get it working it will be extremely useful. I've downloaded your demo code and changed port numer to 20008 as you suggested, however I'm still receiving this error message:

"The file "http://localhost:20008/1.0.0/control-room/3/0/6.png" is missing or inaccessible, make sure the URL is valid or that the file has been added to your sketch and is readable."

The cities load fine, but no tiles. I've also tried using port 20009 but to no avail. Have tried this on both mac and windows but the result is the same.

Hope someone can help.

aart.rost

Re: Unfolding maps, tilemill and streaming tiles: howto

1 year ago

Hi Sirano,

You are using the old StreamProvider class. That one returns URL's for the old version of TileMill.

I noticed that UnfoldingMaps is not compatible with the new (2.0) version of Processing. So please use the 1.5.1 series of Processing. I've updated the example to work with the latest version of TileMill. Download here: http://aartrost.nl/dev/processing/unfolding/streamingMapDemo_v2.zip

Aart.

edodge

Re: Unfolding maps, tilemill and streaming tiles: howto

1 year ago

I'm trying to implement the method that streams from a generated .mbtiles file, but I'm getting the following error:

Cannot find anything named "MBTilesLoaderUtils"

Isn't MBTilesLoaderUtils supposed to be included in the Unfolding library? Any ideas what the problem might be?

Thanks

aart.rost

Re: Re: Unfolding maps, tilemill and streaming tiles: howto

1 year ago

Hi Edodge,

You've got to import the MBTilesLoaderUtils class. You can do that in this way:

import de.fhpotsdam.unfolding.tiles.MBTilesLoaderUtils;

If that still does not work, this is for loading most of the library components:

import de.fhpotsdam.unfolding.geo.*;

import de.fhpotsdam.unfolding.core.*;

import de.fhpotsdam.unfolding.utils.*;

import de.fhpotsdam.unfolding.events.*;

import de.fhpotsdam.unfolding.providers.*;

import de.fhpotsdam.unfolding.mapdisplay.AbstractMapDisplay;

import de.fhpotsdam.unfolding.tiles.MBTilesLoaderUtils;

import de.fhpotsdam.unfolding.geo.MercatorProjection;

edodge

Re: Unfolding maps, tilemill and streaming tiles: howto

1 year ago

Great, thank you!

I'm still having some problems with implementing the map in Processing. I'm new to Processing so I apologize for the basic questions.

Limiting Panning:

I created a map of a single country (Indonesia) in Tile Mill. When I interact with the map in Processing it allows me to pan all the way off the map until all that shows is black. Is there a way to fix this so that panning is disabled if the user tries to go off the map?

I also want to design the sketch so that panning is only enabled by clicking and dragging on certain regions of the sketch.

Zoom range:

When I exported the .mbtile file I specified zoom levels 4-10, and in my setup object I specify:

map.setZoomRange(4, 10);

However, I am still able to zoom outside this range in the Processing sketch. Could another section of the code be conflicting with this command?

Interacting with map regions

In Tile Mill I used the publicly available administrative data to map the district borders of Indonesia. I want to support interaction in my Processing sketch that will show district-level statistics on a mouse hover over that district. I think I need to use the same polygon information in the administrative data shape file along with the map.getScreenPositionFromLocation command. Can you put me on the right path for working this out? Specifically, how do I take the polygon data and turn it into something that Processing will understand?

Thanks!

edodge

tillnm

Re: Re: Unfolding maps, tilemill and streaming tiles: howto

1 year ago

In the latest SVN version there is a method for limiting the panning:

setPanningRestriction(location, maxPanningDistance)

See http://code.google.com/p/unfolding/source/browse/trunk/unfolding/examples/de/fhpotsdam/unfolding/examples/interaction/PanConstrainedMapApp.java for an example.

If you want to use this in Unfolding 0.8, you can use the following:

private void restrictMap(Location restrictedPanLocation, float maxPanningDistance) {
Location mapCenter = map.getCenter();
double dist = GeoUtils.getDistance(restrictedPanLocation, mapCenter);
if (dist > maxPanningDistance) {
float angle = PApplet.degrees((float) getAngleBetween(restrictedPanLocation, mapCenter));
float backDist = maxPanningDistance - (float) dist;
// Pan back, with same angle but negative distance
Location newLocation = GeoUtils.getDestinationLocation(mapCenter, angle, backDist);
map.panTo(newLocation);
}
}
public static double getAngleBetween(Location location1, Location location2) {
double rlat1 = Math.toRadians(location1.getLat());
double rlat2 = Math.toRadians(location2.getLat());
double rlon1 = Math.toRadians(location1.getLon());
double rlon2 = Math.toRadians(location2.getLon());
double angle = (Math.atan2(Math.sin(rlon2 - rlon1) * Math.cos(rlat2),
Math.cos(rlat1) * Math.sin(rlat2) - Math.sin(rlat1) * Math.cos(rlat2) * Math.cos(rlon2 - rlon1)) % (2 * Math.PI));
return angle;
}

and call the method restrictMap() before you draw your map. The distance is in kilometers.

tillnm

Re: Unfolding maps, tilemill and streaming tiles: howto

1 year ago

Regarding the second part on interactive regions. Firstly, I'd recommend converting the Shapefile into some format easier to read (e.g. KML or GeoJSON), for instance with the GDAL tool.

Secondly, you need to parse those yourself, and import them into your sketch. See for example http://tillnagel.com/2009/12/processing-geo-information-in-wikipedia-articles/ on how to read geo-information from an XML file.

Then, I'd recommend using some existing polygon class (e.g. toxic's Polygon2D), and use the coordinates as latitude and longitude. Then, you can use those for contain-tests. In extreme cases, you will face some inaccuracies (due to the Mercator projection), but for most, as for your Indonesia use-case, this will be sufficient.

Also, to draw them simply use beginShape() and vertex() etc.

edodge

Re: Unfolding maps, tilemill and streaming tiles: howto

1 year ago

I'm getting pretty close here but am still running into some problems. Any help would be amazing.

I converted the shapefile containing the province polygons into a json and imported it into Processing. For simplicity at this point I'm using a truncated version of the json with just one province.

Here is the code I'm using to extract the coordinates and build the polygon "prov". Aside from a small issue with data types (floats truncate the coordinate values in the json), the final print statement is giving me the correct output for the polygon "prov".

public void setup() {
Polygon2D prov = new Polygon2D();
try {
JSONObject prov_poly = new JSONObject(join(loadStrings("province2.txt"), ""));
JSONArray entries = prov_poly.getJSONArray("features");
JSONObject features = entries.getJSONObject(0);
JSONObject geometry = features.getJSONObject("geometry");
JSONArray coordinates1 = geometry.getJSONArray("coordinates");
JSONArray coordinates2 = coordinates1.getJSONArray(0);
println( prov );
for (int i = 0; i < coordinates2.length(); i++) {
JSONArray vertex = coordinates2.getJSONArray(i);
double vertexd_x = vertex.getDouble(0);
float vertex_x = (float)vertexd_x;
double vertexd_y = vertex.getDouble(1);
float vertex_y = (float)vertexd_y;
Vec2D p = new Vec2D(vertex_x, vertex_y);
prov.add(p);
}
}
catch (JSONException e) {
println (e.toString());
}
println( prov );
}

Then in the draw method I include the following contain-test after the map is drawn. However, I'm getting the error "Cannot find anything named prov" even though the polygon "prov" was created correctly in the setup method. Any ideas as to why this variable is disappearing? Thanks!

public void draw {
Location mouseLocation = map.getLocationFromScreenPosition(mouseX, mouseY);
float mouseLat = mouseLocation.getLat();
float mouseLon = mouseLocation.getLon();
Vec2D pnow = new Vec2D(mouseLon, mouseLat);
if (prov.containsPoint(pnow)) {
text("North Sumatra", mouseX, mouseY);
}
}

aart.rost

Re: Re: Unfolding maps, tilemill and streaming tiles: howto

1 year ago

Hi Edodge,

You've declared prov inside your setup function. That means prov will only be available within that function. If you wish to use prov as a global var (so you can use it everywhere in the PApplet instance) you'd have to do this:

Polygon2D prov;

void setup(){

prov = new Polygon2D();
}

edodge

Re: Unfolding maps, tilemill and streaming tiles: howto

1 year ago

Thanks!

edodge

Re: Unfolding maps, tilemill and streaming tiles: howto

1 year ago

I'm trying to convert the coordinates from my json file into latitude and longitude for use with the Unfolding library. A typical set of coordinates in my file looks like this:

[ 5584460.799134, 930951.404138 ]

and my .prj file says this:

PROJCS["Mercator_2SP",GEOGCS["GCS_unnamed ellipse",DATUM["D_unknown",SPHEROID["Unknown",6378137,0]],PRIMEM["Greenwich",0],

UNIT["Degree",0.017453292519943295]],PROJECTION["Mercator_2SP"],

PARAMETER["standard_parallel_1",0],PARAMETER["central_meridian",0],

PARAMETER["false_easting",0],PARAMETER["false_northing",0],UNIT["Meter",1]]

--which tells me that the units are in meters. Any ideas on how to do the conversion from (x,y) in meters to (lat, lon)?

Thanks!

Eric

edodge

Re: Re: Unfolding maps, tilemill and streaming tiles: howto

1 year ago

To be more precise, the formula given on Wikipedia is not giving me lat/lon coordinates that line up with the coordinates on the drawn map.

The latitude coordinates on the map are between 2 and 6 for North Sumatra. Applying the y-->latitude formula on North Sumatra's polygon vertices gives latitude coordinates in the 8 to 12 range. I think this may be happening because of the Mercator transformation done in the customMBTilesMapProvider() method:

public customMBTilesMapProvider() {
super(new MercatorProjection(26, new Transformation(1.068070779e7f, 0.0f, 3.355443185e7f, 0.0f,
-1.068070890e7f, 3.355443057e7f)))
}

Any ideas on how to address this so the two coordinate systems line up?

edodge

Re: Unfolding maps, tilemill and streaming tiles: howto

1 year ago

I thought I would share the code I wrote to create linking effects with a map using polygon data. The code below displays the province name on a mouse hover. Hopefully this is helpful to people trying to implement something similar in Processing.

I converted my Indonesian province shapefile into a json, then parsed the polygon information in Processing using the JSON4Processing library. Provinces may contain more than one polygon (eg multiple islands in the same province), so I created an array list of polygons for each province, then nested each province inside a main array list:

public void setup {
try {
JSONObject json = new JSONObject(join(loadStrings("province.txt"), ""));
JSONArray provinces = json.getJSONArray("features");
// start the loop over provinces here
for (int k = 0; k < propPovTable.numRows; k++) {
// get the json observation number that you recorded in the province table
int idpoly = (int)propPovTable.getFloatAt(k,1)-1;
JSONObject pObject = provinces.getJSONObject(idpoly);
JSONObject geometry = pObject.getJSONObject("geometry");
JSONArray coordinates1 = geometry.getJSONArray("coordinates");
ArrayList<Polygon2D> province = new ArrayList<Polygon2D>();
// Collect all the polygons for each province
for (int i = 0; i < coordinates1.length(); i++) {
//make a new Polygon2D for this province
Polygon2D poly = new Polygon2D();
//Multi-polygon case
if (coordinates1.length()>1) {
JSONArray coordinates2 = coordinates1.getJSONArray(i);
coordinates5 = coordinates2.getJSONArray(0);
//Single polygon case
} else {
coordinates5 = coordinates1.getJSONArray(i);
}
// Collect all the vectors for each polygon
for (int j = 0; j < coordinates5.length(); j++) {
JSONArray vertex = coordinates5.getJSONArray(j);
double vertexd_x = vertex.getDouble(0);
double vertexd_y = vertex.getDouble(1);
float vertex_x = (float)vertexd_x;
float vertex_y = (float)vertexd_y;
//convert from meters to lat/lon coordinates
double vertex_x_coord = vertexd_x / earthRad * RADtoDEG;
float lon = (float)vertex_x_coord;
float vertex_y_coord = (2*atan(exp(vertex_y/earthRad)) - (PI/2))*RADtoDEG;
float lat = (float)vertex_y_coord;
Vec2D p = new Vec2D(lat, lon);
poly.add(p); // add p to the polygon poly
}
province.add(poly); // add polygon poly to the province polygon list
}
polygonListAll.add(province); // add the province polygon list to the main array list
}
}
catch (JSONException e) {
println(e.toString());
println(idpoly);
}
}

Then in the draw method I detect the mouse position and link it to the polygon positions on the map:

public void draw() {
Location mouseLocation = map.getLocationFromScreenPosition(mouseX, mouseY);
float mouseLat = mouseLocation.getLat();
float mouseLon = mouseLocation.getLon();
Vec2D pnow = new Vec2D(mouseLat, mouseLon);
for(ArrayList<Polygon2D> p:polygonListAll) {
for(Polygon2D q:p) {
if (q.containsPoint(pnow)) {
//display the province name
textAlign(RIGHT);
text(propPovTable.getDataAt(polygonListAll.indexOf(p),2), width-30, 110);
}
}
}
}

filococus

Re: Unfolding maps, tilemill and streaming tiles: howto

1 year ago

Hi edodge,

Currently i'm trying to do the same but i'm new to geojson files and toxiclibs!

I want to know how do you visualize the geometry not just the name of the province. It will be very helpful if you can upload the full example because the code above seems to be incomplete (some missing objects such as propPovTable or coordinates5) so it's difficult to reproduce this example.

Thanks for sharing!

edodge

Re: Re: Unfolding maps, tilemill and streaming tiles: howto

1 year ago

Hi filococus,

Once you have the geometry as polygons you can use beginshape() and vertex() to draw them to the screen.

propPovTable is a dataset that contains my universal province id's, the observation number of that province in the json file (I got these by browsing the .dbf file), and the province name:

id idpoly name

11 1136 Aceh

12 381 Sumatera Utara

13 539 Sumatera Barat

14 492 Riau

15 1928 Jambi

16 1230 Sumatera Selatan

17 1225 Bengkulu

18 1229 Lampung

19 1231 Bangka Belitung

21 1796 Kepulauan Riau

you will need to make a similar dataset as well as initialize these objects:

float RADtoDEG = 57.295779513082322;
float earthRad = 6378137;
int idpoly;
ArrayList<Polygon2D> polygonListProv = new ArrayList<Polygon2D>();
ArrayList<ArrayList<Polygon2D>> polygonListAll = new ArrayList<ArrayList<Polygon2D>>();
JSONArray coordinates5;

Hope this is helps!

filococus

Re: Unfolding maps, tilemill and streaming tiles: howto

1 year ago

Wow nice! it worked! i'm not sure that it is the most efficient way but....

This is what i did in the draw method! (I changed some variable names)!!!

public void draw() {
map.draw();
fill (255,255,0);
stroke(255,0,0);
for(ArrayList<Polygon2D> p:polygonListAll) {
for(Polygon2D singlePoly:p) {
beginShape();
List<Vec2D> verticesList = singlePoly.vertices;
for (Vec2D singleVertex : verticesList){
float [] newPos = map.getScreenPositionFromLocation(new Location (singleVertex.x, singleVertex.y));
vertex(newPos[0],newPos[1]);
}
endShape();
}
}
}