Add latitude/longitude to csv file and make interactive map with the data

Dirma

Hi everyone,

I'm quite a beginner in p5.js and I want to do the following:

I have a csv table with data of medication use in The Netherlands, per town. This data I want to show on an interactive (mapbox) map.

Problem 1:
The csv file only has the names of the towns, not the longitude & latitude. Is there a way to generate this automatically? Otherwise I have to do it manually for about 400 towns:(.

Problem 2:
When I have the longitude and latitude, how can I show them on a map, using mapbox, via p5.js?

I hope you can help me with this!!!

Best,

Dirma

kfrajer

Check this post:
https://forum.processing.org/two/discussion/20910/slippy-maps-on-p5-js#latest

And in this next code, you can add the cities to the array and it will load them in the map: http://p5js.sketchpad.cc/sp/pad/view/R14cPjGmuo/latest

Hover the mouse over the cities to display their names. I have a blinking effect at each location, in case you are wondering why the yellow circles are changing over time.

Kf

Dirma

Hi kfrajer,

Thanks for the help, really appreciated. I don't understand the following code:

for(let cc of cities){ let readGeo =loadJSON('https://'+'maps.google.com/maps/api/geocode/json?address=$'+cc+'&sensor=false&region=Netherlands'); geoData.push(readGeo); } }

What do 'let' and 'cc' mean?. And how can I load the names of the places into the 'var cities' array? or do I need to add d=them manually?

Best,

Dirma

kfrajer

You can add them manually or you can read a file or URL using loadStrings. You can check the P5.js reference for an explanations and check the provided examples. If it is not clear, ask below and any of the forum goers will be happy to provide you an example or to point you to a previous post with one.

So as you see in the link, what I am doing is this:

1. I start with an array of know cities ( I don't know many places in Netherlands... sorry)
2. In preload, I inquire for all the geoinformation of those cities. I am not sure if you know this, but preload executes all its statements and only after all statements are done, the program continues its normal execution aka. setup() will be executed follow by continuous calls to draw().
3. Then in setup, I retrieve my geo object and I extract all the information I need like lat and lon.

The push() function is associated to array objects. I am pushing or adding some specific info to my array. This is the way to do so.

This code was just a simple demonstration of how to accomplish your task. It can be adapted to your purpose and isntead of preload, you could use a callback method. I recommend checking the video I provided in the previous post as @shiffman does a terrific job explaining the usage of mapbox and how to interact with it.

Kf

GoToLoop

1. let: https://developer.Mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/let
2. for...of: https://developer.Mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/for...of

Dirma

Thanks again for the help, I'll check it out today!!

Dirma

Hi,

I'm still dealing with a couple of problems..: I tried it in 2 ways:

1: I used loadStrings, but the places don't show on the map:

`"use strict";  

 var result;
  var map;
  var zoom=6.44;
  const kMaxR=25;
  const kMinR=5;

  var geoData=[];
  var clat=51.949516;
  var clon=6.060097;
  var cc;
  var cx;
  var cy;

  var cities;
  var lat_cities=[];
  var lon_cities=[];
  var rad=kMaxR;

  const TEXT_SIZE = 18, FPS = .5; 


  function preload(){
      cities = loadStrings('assets/citiesNL.txt');

    map=loadImage('https://'+'api.mapbox.com/styles/v1/dirma/cizi74roy001h2spanpymjfdd/static/'+clon+','+clat+','+zoom+',0.00,0.00/1200x800?access_token=pk.eyJ1IjoiZGlybWEiLCJhIjoiY2lpMDl1M3doMDR1NXVza29vanVmOW9nYSJ9.hB2cpg0ETqpqa6MGHU6qIw');    

    for(let cc of cities){
           let readGeo =loadJSON('https://'+'maps.google.com/maps/api/geocode/json?address=$'+cc+'&sensor=false&region=Netherlands');      
      geoData.push(readGeo);      
    }    
  }



  function setup() {  

    createCanvas(1200,800);
    stroke(random(50), random(255), random(255), 100);
    background(255);
    frameRate(FPS).background(0).noStroke();
    textAlign(LEFT, TOP).textSize(TEXT_SIZE);
    imageMode(CENTER);

    for(let readGeo of geoData){
      let res=readGeo.results;
      let tok=res[0].geometry.location.lat;
      lat_cities.push(webmercatorY(tok));
      tok=res[0].geometry.location.lng;
      lon_cities.push(webmercatorX(tok));
     }

     //Map's center
      cx = webmercatorX(clon);
      cy = webmercatorY(clat);
  } 


  function draw() {          

    translate(width/2,height/2);      
    image(map,0,0);    

    for(var j=0;j<lat_cities.length;j++){

      //Cities location on canvas
      var x = lon_cities[j] - cx;
      var y = lat_cities[j] - cy;

      fill(220,20,220,180);
      ellipse(x,y,kMinR,kMinR);

      rad--;
      if(rad<kMinR) rad=kMaxR;

      if(dist(mouseX-width/2,mouseY-height/2,x,y)<kMaxR){
          text(cities[j]+'',x,y);
      }          
    }
  }


  //longitude in degrees
  function webmercatorX(longitude){
      longitude=radians(longitude);
      var a = (256/PI)*pow(2,zoom);  //Mapbox uses 512x512 tiles, so half is 256
      var b = longitude + PI;
      return a*b;
      }


  //latitude in degrees
  function webmercatorY(latitude){
      latitude=radians(latitude);
      var a = (256/PI)*pow(2,zoom);  //Mapbox uses 512x512 tiles, so half is 256
      var b = tan(PI/4+latitude/2);
      var c = PI - log(b);
      return a*c;
      }`

2: I added the places manually, but for some reason it doesn't show some places of the cities array. Some places seem to give errors, for example 'Ameland'. When I put a place like 'Ameland' or 'Aalsmeer' in the array, sometimes the map shows and sometimes I get this error: 'Uncaught TypeError: Cannot read property 'geometry' of undefined'. Does anyone know what I'm doing wrong? :

`"use strict";  

  var map;
  var zoom=5.62;
  const kMaxR=25;
  const kMinR=5;

  var geoData=[];
  var clat=52.172855;
  var clon=5.917137;
  var cc;
  var cx;
  var cy;

  var cities =new Array(
 "Aa en Hunze",
    "Aalburg",
    "Aalsmeer",
    "Aalten",
    "Achtkarspelen",
    "Alblasserdam",
    "Albrandswaard",
    "Alkmaar",
    "Almelo",
    "Almere",
    "Alphen aan den Rijn",
    "Alphen-Chaam",
    // "Ameland",
    "Amersfoort",
    // "Amstelveen",
    // "Amsterdam",
    // "Apeldoorn",
    // "Appingedam",
    // "Arnhem",
    // "Assen",
    // "Asten",
    // "Baarle-Nassau",
    // "Baarn",
    // "Barendrecht",
    // "Barneveld",
    "Bedum",
    "Beek (L.)"
    // "Beemster"
    // "Beesel",
    // "Bellingwedde",
    // "Bergambacht"
   );  
  var lat_cities=[];
  var lon_cities=[];
  var rad=kMaxR;

  const TEXT_SIZE = 18, FPS = .5; 


  function preload(){

    map=loadImage('https://'+'api.mapbox.com/styles/v1/kfrajer/cizvxbzd900102sllt7lozcad/static/'+clon+','+clat+','+zoom+',0.00,0.00/1200x800?access_token=pk.eyJ1Ijoia2ZyYWplciIsImEiOiJjaXp2eDc2eHUwMWVtMnBxcW1jcnUzZjdxIn0.W9vZESizqDyxC85hfeoHeQ');    

    for(let cc of cities){

      let readGeo =loadJSON('https://'+'maps.google.com/maps/api/geocode/json?address=$'+cc+'&sensor=false&region=Netherlands');      
      geoData.push(readGeo);      
    }    
  }



  function setup() {  

    createCanvas(1200,800);
    stroke(random(50), random(255), random(255), 100);
    background(255);
    frameRate(FPS).background(0).noStroke();
    textAlign(LEFT, TOP).textSize(TEXT_SIZE);
    imageMode(CENTER);

    for(let readGeo of geoData){
      let res=readGeo.results;
      let tok=res[0].geometry.location.lat;
      lat_cities.push(webmercatorY(tok));
      tok=res[0].geometry.location.lng;
      lon_cities.push(webmercatorX(tok));
     }

      cx = webmercatorX(clon);
      cy = webmercatorY(clat);
  } 


  function draw() {          

    translate(width/2,height/2);      
    image(map,0,0);    

    for(var j=0;j<lat_cities.length;j++){


      var x = lon_cities[j] - cx;
      var y = lat_cities[j] - cy;

      fill(220,20,220,180);
      ellipse(x,y,kMinR,kMinR);

      rad--;
      if(rad<kMinR) rad=kMaxR;

      if(dist(mouseX-width/2,mouseY-height/2,x,y)<kMaxR){
          text(cities[j]+'',x,y);
      }          
    }
  }


  function webmercatorX(longitude){
      longitude=radians(longitude);
      var a = (256/PI)*pow(2,zoom);  //Mapbox uses 512x512 tiles, so half is 256
      var b = longitude + PI;
      return a*b;
      }


  function webmercatorY(latitude){
      latitude=radians(latitude);
      var a = (256/PI)*pow(2,zoom);  //Mapbox uses 512x512 tiles, so half is 256
      var b = tan(PI/4+latitude/2);
      var c = PI - log(b);
      return a*c;
      }`

kfrajer

Please edit your post, select your code and hit ctrl+o to format your code. Make sure there is an empty line above and below your code

Kf

Dirma

Ah ok, thanks, now it looks readable:).

kfrajer

For number 1, is your file being read correctly? If it is, then the problem is number 2.

Number 2 just means the city was not found (probably). In the example, I didn't provide any error check. However, by the description you gave above, that is exactly what is happening:

Uncaught TypeError: Cannot read property 'geometry' of undefined

I can look into this later in the day or maybe tomorrow. It is important you make sure you are able to access your data file for cities though.

Also, are you using your own auth key? I think my could have expired. Unfortunately I cannot check this right now.

Kf

GoToLoop

https://forum.Processing.org/two/discussions/tagged?Tag=loadjson()

kfrajer

Ok, so I tried to access the json data directly like this:

maps.google.com/maps/api/geocode/json?address=$netherlands&sensor=false&region=Netherlands

http://maps.google.com/maps/api/geocode/json?address=$Aalsmeer&sensor=false&region=Netherlands

http://maps.google.com/maps/api/geocode/json?address=$Ameland&sensor=false&region=Netherlands

and it works. Also mapbox's key is still valid. I tried the following and it works:

var cities =new Array(
    "Aa en Hunze",
    "Aalburg",
    "Aalsmeer",
    "Aalten",
    "Achtkarspelen",
    "Alblasserdam",
    "Albrandswaard",
    "Alkmaar",
    "Almelo",
    "Almere",
    "Alphen%20aan%20den%20Rijn",
    "Alphen-Chaam",
    "Ameland",
    "Amersfoort",
    "Amstelveen",
    "Amsterdam",
    "Apeldoorn",
    "Appingedam",
    "Arnhem",
    "Assen",
    "Asten",
    "Baarle-Nassau",
    "Baarn",
    "Barendrecht",
    "Barneveld",
    "Bedum",
    //"Beek (L.)"
    "Beemster",
    "Beesel",
    "Bellingwedde",
    "Bergambacht"
   ); 

Notice I replaced in some entries spaces with %20. However it seems that is not a problem. There is no problems with hyphens (-) either but I think parentheses will need special treatment. Related links:

http://stackoverflow.com/questions/32348875/can-hyphens-be-used-in-query-string-values
https://forum.processing.org/one/topic/how-make-urlencode.html
http://stackoverflow.com/questions/332872/encode-url-in-javascript

A an alternative, I did the following change when requesting the JSON file for the city which also works:

var acity=encodeURI("https://maps.google.com/maps/api/geocode/json?address=$"+cc+"&sensor=false&region=Netherlands");
let readGeo =loadJSON(acity); 

Here is the latest sketchPad version: http://p5js.sketchpad.cc/sp/pad/view/R14cPjGmuo/latest

In regards to parentheses in one of the names of your cities, you should consider revising a different version or you need to look into the documentation: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/encodeURI

You need to test loading files from external resources. Try loading a simple file for starters or post the code that you are using and the description of the location of your assets. This post might provide better insight: https://forum.processing.org/two/discussion/13393/reading-api-problem-cannot-read-property-responsetext-of-undefined#latest

Kf

Dirma

Hi kfrajer,

Thanks again for helping:)

It works! glad to be one step further in this project. So I copied the code of http://p5js.sketchpad.cc/sp/pad/view/R14cPjGmuo/latest? and put the whole array in the sketch (409 cities). The strange thing is that it does load the cities on the map, but not the whole array. Every time I reload the sketch it loads a different selection. See these images:

Is there in the code a maximum for the cities on the map? I couldn't find it..

Best,

Dirma

kfrajer

Not sure what could cause that. One way to go about it is to print out how many entries are being stored in the json container when preload ends. Another way is to optimize the process creating a file that stores lat/lon values. Every time that you load the program, you inquire for lat/lon of cities that are not stored in the file.

Kf

jeremydouglass

Is there in the code a maximum for the cities on the map?

Without examining your code carefully, the most likely limit is from an API.

for(let cc of cities){
  let readGeo =loadJSON('EACHCITYURLGOESHERE');      
  geoData.push(readGeo);      
}

Perhaps you hit maps.google.com with a bunch of requests (too many) all at the same time (too fast) -- it answers some, and doesn't answer others. Then, when you display, different things are missing each time.

koogs

(That's mangled the URL so badly that I don't know how to fix it)

kfrajer

@koogs

This is the link http://maps.google.com/maps/api/geocode/json?address=$Aalsmeer&sensor=false&region=Netherlands

where the address value is changed based on the city item in the list.

Kf

Dirma

Yes, I think it's the limit of the Google API:

2,500 free requests per day, calculated as the sum of client-side and server-side queries. 50 requests per second, calculated as the sum of client-side and server-side queries.

Maybe it only shows the 50 requests of the first second?

But then it's best to create a file with the lat/lon values I think.

Best, Dirma

jeremydouglass

it's best to create a file with the lat/lon values I think.

Absolutely.

For example, write a separate NetherlandsCityLookup sketch that builds a location cache file -- it submits say 25 requests per second (frameRate(1)) and saves them to a csv file using Table / saveTable. Then have your Medication sketch load that data using loadTable.

Later you can combine those into one sketch -- your Medication sketch has an "update cities" function that it can run which will build a new lat/lon cache file. However, because cities don't change GPS coordinates very often (!!), you won't need to run it unless your city list changes -- which means your Medication sketch can run offline and doesn't rely on the Google API (which might change or go away).