Articles

10.4: Loading JSON data from a URL (Asynchronous Callbacks!) – p5.js Tutorial


In this video, I want to
look at loading JSON data not from a local file– like a file like birds.json
that’s right there in your project– but from a URL. And this is one
step along the way to querying an API, to saying
like, hey, weather map thing, give me some weather data. Or, hey, New York Times, give
me some New York Times data. But I want to start
simpler than that. So, first of all, this is
actually really, really easy, in the sense that
all I need to do is replace birds.json with a
URL that gives me JSON data. But it brings up a
question about how you work with loading data
beyond just this preload thing. So the preload idea
was super convenient. It guarantees that
all the data is loaded before the program starts. So you’ve got all the data. You could do anything
you want with it as the program’s running. But this breaks down very
quickly and very easily. So, for example,
here’s a scenario where you want to have some
sort of weather visualization, like you’re going to show
the sun if it’s sunny out, or some rain if it’s raining. And you have a button
that just says Load Data. So every time the user
presses that button, it should get the most recent
data, the current weather. So you can’t use
preload for that because you can’t
preload the future. Boy, wouldn’t that be great? So when you invent the
whole time travel thing, you can use preload. Like preload time– maybe
that’s how we get time– I don’t know. But you can see how this
doesn’t work, right? You can’t preload the data. You’ve got to load the data
while the program is running. Certainly, you could
also make the same case of, as a program is
running, you just want to continuously– like
if you’re trying to visualize financial data and it’s
just changing every three seconds, in your draw loop, you
might want to query the data, and query the data,
and query the data. You also don’t want the program
to stop animating while it’s waiting to load the data. So you need to load
the data in what’s known as an asynchronous manner,
meaning when the data can arrive at another sort
of– in another sort of like parallel universe
of your code that’s running. So your code is trucking along. It’s starting, it’s
animating, it’s looping, it’s doing its stuff. And at any point there, it
could say, hey, load some data and let me know when
the data is loaded. So how do you do that? We said previously in preload,
var birds equals loadJSON. This is the synchronous way. Preload loads the data the
program waits and waits and waits and waits to start. The asynchronous way
is to just simply only call the loadJSON, function,
pass it the path of the data that you want to load– birds.json or some URL– and then what’s known
as a callback function. So I’m going to call my
callback function gotData. So this, by the way,
is exactly the topic that I discussed in a
previous set of videos all about looking
at HTML elements, like buttons and
sliders, and what happens if you have a callback
when you click the button. So I’ll try to link back
to the video about callback and events. This is exactly
the same idea here. I now need to write
a function to handle the asynchronous nature
of the data arriving later at some other time, separate
from my program that’s running in its animation loop. So if I say function
gotData, this function is the callback for
loading the data. I’m starting a process. Hey, go and load some data. And when you’re
done, let me know. The let me know is
trigger the code in here. Now, this isn’t magic. This isn’t some default
thing that just happens. This is the nature of how just
about everything in JavaScript works. But loadJSON and this
methodology, this is specific to
the p5 JS library. Of course, with
native JavaScript, or jQuery, or all
these other frameworks, they all have something
similar to this. So, conceptually,
this is a concept that’s across the board
in working with data. But the syntax and this function
name is specific to p5 JS. So I give it the URL. I give it the callback. And then this
function is triggered when the data is ready. Like, the data is ready. I got to sing again. That’s good. The question is,
where is the data? Like I’ve completely
lost– like here, the data is in this
variable called birds. That’s very obvious because I’m
saying birds equals loadJSON. But there’s no
variable anywhere. There’s just the URL,
the loadJSON function, and the callback. So the special
way that p5 works, and that a lot of things
in JavaScript works, is if you add a parameter
to your function definition, your callback definition– I’m calling it data– p5, behind the scenes,
will fill that variable in the function with the data. So it’s like you said
this data variable equals the result of loadJSON, but
it happens asynchronously. loadJSON from this URL. When you’re done,
trigger this function. When that function’s
triggered, the data will be in that variable. Got it? I don’t know what to do. That’s all I can say about it. But send me your messages. And write them and
get a carrier pigeon, or Twitter, or whatever, email. Contact me, say hi. I need– human interaction is
a useful and important thing. So let’s make that
happen over here. So, in other words, instead of
saying data equals loadJSON, I want to say– and I don’t
want to have preload, actually, at all. Like I could actually–
just here, I could say– actually, let’s– I could say– I’ll just put it in setup. I’ll make this more
interesting later. I could just say
loadJSON birds.json. They’re coming for me. Help, everybody. My heart is racing a little
bit, so maybe somebody knows that I might need
some medical attention. birds.json is the
name of the file. gotData is going to be the
name– that’s a name that I’m making up for the callback. I’m going to create that
function called gotData. I’m going to give it a
parameter right there. So now this is the process. Start the process
of loading the data. When it’s loaded,
execute this function, fill this variable with data. And now I can say,
console.log data. Oh, sorry. println– I like to use the– it doesn’t matter. But I just want to
see that this works. And I’m going to run this. You can see that’s
happening, and it’s happening super fast
because the data is loaded and then this is triggered. Great. It works. So now let’s move on
from this birds data set and let’s think about
what it means now to pass a URL into
that function and how we might query the data at
a particular moment in time while the program is running. So what I want to do
is I’ll use a simple– an API that I like to use,
which is a URL for JSON data that I like to use, which
tells us how many people are in space right now. So I’m going to go grab
this URL right here. I’m going to paste
it into the browser just to see that this works. And you can see– look at this. This URL is giving me JSON data. What is it giving me? It’s giving me information about
the number of people in space. And you can see it has– it’s
an object with the number 6. And it’s got, then,
an array of people, with their craft, the
International Space Station, and their name. So the number of people
currently in space is 6, and these are all
the people and that’s the spaceship that they’re on. Now, ideally, it would be great
if this was a data set that changed very often,
so if I hit refresh, it would say 7, and
then 8, and then 4. But I guess people aren’t like
going to and from space very often. So this isn’t necessarily
the best demonstration of why you would need this. But in some future
videos, I’ll get to data that changes and that needs to
be refreshed in your program. So all I need to do
now, if this works, is take this particular URL
and paste it right in here. So you can see
that loadJSON works not just with a local file
but with a particular URL. And now, if I run this– oh, I don’t see anything. I wonder– hmm, interesting. And I didn’t even get an error. I’m going to just add something. Close your eyes and don’t pay
attention to what I’m doing. Yes, it worked. So this is a good moment
to have this discussion. Security is a thing. Authenticating with
your name and a password and allowing a browser– your browser and your machine
to talk to your camera or to talk to another server. Working with data
opens up this big can of worms of what
you’re allowed to do and what you’re
not allowed to do. And there comes a
time in your life where you’ll find a
URL, and that URL, you know it gives you JSON
data because I pasted it into the browser right there
and I see that JSON data right there. But I put it in my loadJSON
function and nothing happened. Now, honestly, I didn’t
even get an error here. There’s a bit of a
secret that I think I’ll show you, which is, right
up here in the browser window, there’s this settings. And I can click on that and
I can get some more error messages. And you can actually see
the error message that came. XMLHTTPRequest cannot load. There’s no
Access-Control-Allow-Origin header is present. Origin is therefore
not allowed access. This error message is
the bane of my existence. I always forget that
it might happen, and sometimes it doesn’t happen
and sometimes it does happen. This has nothing to
do with you or me. This has to do with
how the server, the place that the URL
address is configured. And this URL does
not necessarily just allow you by default
to grab the data. However, there is
something called JSONP. The P stands for Padding. I like to think of
it like a safety net. It’s like I’m in a
room with padded walls and I won’t hurt myself
asking for the JSON data. And I feel like that would
be an interesting topic for another video, to delve
into that more deeply. So, sometimes, if you’re
not allowed access in the sort of
traditional way, you can use this thing called
JSONP to get the data. And you know that one
of the ways that I do is I put the URL in loadJSON,
and when it doesn’t work I just add JSONP and I hope it works. And, usually, it’s going to– I tried first without it. If it doesn’t work,
I try it with it. You can write your– I’ll maybe include a link to
more information about JSONP in the description’s video. So what do I mean by adding it? So it’s an optional
third argument to the loadJSON function. So the loadJSON function
requires a URL and a callback. And, optionally, you can
add a third argument, which is the string
jsonp, which allows you to get past this
access origin control crazy error thing that
sometimes comes up, where the server’s not
allowing you to get clean JSON. You need JSON’s little padding. There’s no padding over here. Hurt a little bit. I’m a very sensitive
and fragile person. So anyway. So, fortunately, if I type
jsonp as that third argument– it’s good that that
happened because it’s going to happen to
you– and I run it, you can see that the data
is now arriving there. I’m seeing it. So let’s do something
with this data to finish of this
particular video, that’s already 11 minutes long. So if I look at this data,
one thing I might like to do is just draw a circle for
every single person that is– I’ll draw a circle for
every single person that’s in space right now. So, interestingly
enough, if I know that the data is available
in the gotData function, that’s a place where I
could do some drawing. So, for example, I could
say for var i equals 0, I is less than data.number– that’s the number 6– i plus plus. And I could say ellipse at
random width, random height, 16, 16. I could say fill 255. This is going to be the least
interesting data visualization anyone has ever made
in their entire life. But we can now see– and I
could say createCanvas 200, 200. I can now see that
when I run this, I now get this lovely canvas
with six circles on it. So this is– look. I did data visualization. I loaded a URL. I gave it a callback. I even used JSONP
for some padding. Wait, I have my padding song. But that’s different. That’s like CSS. Anyway. I need a separate video which
s just the padding song. And then I pulled
out the number. I made a loop and I
drew circles based on how many people in space. And if I could just
keep refreshing this, I’ll get it in a
different location. But if somebody would just
leave space right now, then you would see five
circles instead of six. So this is the basic idea. I do think, however, there
are a couple other pieces of this that I want
to demonstrate, although you could stop
watching now if you want. You could stop watching
at any time if you want. So one that I did,
it was kind of nice. Like, this is just static. I don’t need anything
to animate here, so I just put the
drawing stuff in gotData. But let’s say what you
actually wanted to do was have a particular
animation that’s going on. So you want your–
like, for example, I want the background– I want to have– this isn’t going
to be interesting, but let’s say I want
to have this line that goes across the screen
from x to 0, x to height. x equals x plus 1. And if x is greater
than width, x equals 0. And let’s say stroke 255. So, in other words,
I want the circles to appear with this animation. So if I were to put the
drawing back in gotData– I know I’m doing
this kind of quickly and scrolling around like a
madman here on the screen– this is not going to work. Because, remember,
draw is looping to always draw that animation. gotData is an event that
happens just once when the data is ready. So that happened and
it drew those circles but it immediately got
overwritten by draw. So there’s an easy solution to
this problem, you might think, which is, OK, just
take the circles and put them here in draw. So now what’s– so there’s
a lot of problems with this. Let me run this. First of all, it’s not working. Why is it not working? Now, hopefully, I probably
have some error message which we could see. Uncaught ReferenceError,
data is not defined. Well of course it’s not defined. I can’t use this variable
data down here and draw. gotData is a callback. That variable data is only
available in that function. So one thing I could
do is I could say var– I’m going to say spaceData. I could make up a global
variable and I could say, spaceData equals data. So this is now– I need a way of– this a sort of simplistic way
of doing it but it will work. I need a way of
connecting the data that came in to something
that’s happening in draw, and a quick way of doing that
is just have a global variable. So I, one, ask for the data. When the data comes
in, I immediately store it in some other
variable, a variable that I can access in draw. So now if I run this– and
you can see me accessing it there in draw. If I run this again– ah, now what’s happening? Oh, sorry. I have to change
this to spaceData. Ah, cannot read property
number of undefined. Oh, what happened? Ah, what’s not working? I know what’s not working. I’m pausing to let you
think about it for a second. Asynchronicity. If that’s even a word, I’m not
sure, but this is the issue. Remember, how are all of these–
what’s the sequence here? Let me try to get everything
here on the screen. Let’s just move this
animation for a second below here so we can see
the pieces that matter. There’s setup, where I’m
asking to load the data. There’s gotData, where
I’m retrieving the data. And there is draw, where
I am using the data. What’s the order of the program? setup starts first. Make the canvas. The canvas is there. Load the data. That’s a process that’s now
happening in the background. Let’s go to draw. Start drawing, draw
with spaceData. The data’s not loaded yet. This is going to take a while. You’ve got to query the
URL, make some connections across the internet,
get the data back. So spaceData is not anything
until this happens, but draw has already started. So there is a nice way
of dealing with this and it has to do with a
nice trick that you can do in JavaScript, which is
using a Boolean test, an if statement, which is a
concept you’re probably quite familiar with. We usually think of
a Boolean expression as something that
evaluates to true or false. Like, if 7 is greater than
5, this is absolutely true. But what does it mean for
me to say if spaceData? Now, one of the nice things
about JavaScript is it just– all it ever wants to do is to
have things be true or false. Like a number, it can be
evaluated to true or false. Like if it’s 0, it’s false. If it’s any other number,
they think it’s true. A string probably
could be evaluated. If it’s the string
false, that’s probably going to evaluate to false. So what is actually
the value of spaceData before the data comes in? It’s undefined. If you have a variable
that’s not initialized– like up here, spaceData is not
initialized– its initial value is undefined. And undefined
evaluates to false. So spaceData– and
defined, like an object, just automatically
evaluates to true. So you can basically
say if spaceData as a way of saying
if spaceData exists. So spaceData, it doesn’t exist
until the data has actually come in. So if I just add that– whoops. Oh, I was in the wrong view. You’re going to have
to live with that. I can write here– I can say if spaceData,
then do this. So now, even though draw starts
looping, this part of draw won’t actually happen until
spaceData gets built here. And now we can see this works. Now, of course, I’m drawing
them at random locations. Here’s actually a really
quick trick for me to get that to stop. I could just say randomSeed. It’s kind of a topic
for another video, but randomSeed is a function
that seeds the random number generator so you always
get the same exact sequence of random numbers. And that’s going to
work for me here, at least just in a sort
of quick and dirty way. So you can see I now have an
animation and a visualization at the same time. I’m not drawing the
part with the data until the data has come in. So I was very long-winded
in this particular video. I could have done
this in half the time. But what I would suggest
to you right now, if you’re looking for an exercise to do,
is go to this particular data source– I’ll link to it below– and see if you can at least
add now, add the people. Like what could you add? The craft and the name as
text next to the circles or something. You’re going to use the
rest of the data here also and quickly get it to work
is this asynchronous callback way. And I think now I’ve
actually covered enough that the next step
it can really be is, what if you’ve got an
API, like the New York Times or Instagram, that’s
serving up data? How do you query that API with
loadJSON and do stuff that way? Boy, this is a 20-minute video. Oh, I don’t know about this.

Leave a Reply

Your email address will not be published. Required fields are marked *