Understand very basics of APIs
Contact an API using R
Process returned data
Goals
APIs
Application Programming Interfaces (APIs) - a defined method for asking for information from a computer
Important for grabbing data, often returned in JSON format
- Very few packages for contacting APIs are out there for R (as compared to say python)
- Can do it yourself using
httrpackage!
Steps to obtain data
Install packages needed for contacting APIs and handling data
httrandjsonlite
(Usually) Obtain a key by registering at the API you want to contact
Construct a URL to obtain data (using
GET())Process data using
jsonlitefunctions
Example https://newsapi.org/
Registered for a key at newsapi.org. An API for looking at news articles
Look at documentation for API (most have this!)
Example URL to obtain data is given
https://newsapi.org/v2/everything?q=bitcoin&apiKey=myKeyGoesHere
Example https://newsapi.org/
- Can add in date for instance:
https://newsapi.org/v2/everything?q=bitcoin&from=2023-05-01&apiKey=myKeyGoesHere
Using R to Obtain the Data
Use
GETfromhttrpackage (make sure to load package!)Modify for what you have interest in!
library(httr)
GET("http://newsapi.org/v2/everything?qlnTitle=Carolina&from=2023-05-01&language=en&
apiKey=myKeyGoesHere")
Returned data
- Usually what you want is stored in something like
content
str(myData, max.level = 1)
## List of 10 ## $ url : chr "http://newsapi.org/v2/everything?qInTitle=Carolina&from=2023-05-01&language=en&pageSize=100&apiKey=aa4b545bfbd6"| __truncated__ ## $ status_code: int 200 ## $ headers :List of 14 ## ..- attr(*, "class")= chr [1:2] "insensitive" "list" ## $ all_headers:List of 1 ## $ cookies :'data.frame': 0 obs. of 7 variables: ## $ content : raw [1:95039] 7b 22 73 74 ... ## $ date : POSIXct[1:1], format: "2023-05-19 03:29:46" ## $ times : Named num [1:6] 0 0.00381 0.01399 0.01411 0.1618 ... ## ..- attr(*, "names")= chr [1:6] "redirect" "namelookup" "connect" "pretransfer" ... ## $ request :List of 7 ## ..- attr(*, "class")= chr "request" ## $ handle :Class 'curl_handle' <externalptr> ## - attr(*, "class")= chr "response"
Parse with jsonlite
Common steps:
- Grab the list element we want
- Convert it to characters with
rawToChar() - Convert it to a data frame (or list) with
fromJSONfrom thejsonlitepackage
parsed <- myData$content %>% rawToChar() %>% jsonlite::fromJSON() str(parsed, max.level = 1)
## List of 3 ## $ status : chr "ok" ## $ totalResults: int 800 ## $ articles :'data.frame': 100 obs. of 8 variables:
Inspecting article info
parsed$articles %>% select(title, description, source, author, everything()) %>% head()
## title ## 1 North Carolina governor vetoes 12-week abortion ban - Reuters ## 2 North Carolina House passes 12-week abortion ban - Reuters ## 3 South Carolina advances 6-week abortion ban - Reuters ## 4 North Carolina lawmakers to vote on overriding veto of 12-week ... - Reuters.com ## 5 Nikki Haley rallies voters in South Carolina ## 6 Fears grow in North Carolina as ultra-extreme Republican eyes governorâ\200\231s mansion ## description ## 1 North Carolina governor vetoes 12-week abortion ban  Reuters ## 2 North Carolina House passes 12-week abortion ban  Reuters ## 3 South Carolina advances 6-week abortion ban  Reuters ## 4 North Carolina lawmakers to vote on overriding veto of 12-week ...  Reuters.com ## 5 Republican presidential candidate Nikki Haley rallied voters during a campaign stop in Greer, South Carolina. (May 5) ## 6 If Mark Robinson wins gubernatorial race in 2024 there would be no one to hold back a wave of rightwing bills in the stateTo Mark Robinson, gay and transgender people are â\200œfilthâ\200\235, homosexuality is an abominable sin, and the transgender movement is â\200œdemonicâ\200\235 aâ\200¦ ## source.id source.name author ## 1 google-news Google News <NA> ## 2 google-news Google News <NA> ## 3 google-news Google News <NA> ## 4 google-news Google News <NA> ## 5 <NA> Yahoo Entertainment Associated Press Videos ## 6 <NA> The Guardian Adam Gabbatt ## url ## 1 https://consent.google.com/ml?continue=https://news.google.com/rss/articles/CBMiN2h0dHBzOi8vd3d3LnJldXRlcnMuY29tL3ZpZGVvL3dhdGNoL2lkT1YyMjY4MTMwNTIwMjNSUDHSAQA?oc%3D5%26hl%3Den-CA%26gl%3DCA%26ceid%3DCA:en&gl=FR&hl=en-CA&cm=2&pc=n&src=1 ## 2 https://consent.google.com/ml?continue=https://news.google.com/rss/articles/CBMiXWh0dHBzOi8vd3d3LnJldXRlcnMuY29tL3dvcmxkL3VzL25vcnRoLWNhcm9saW5hLWhvdXNlLXBhc3Nlcy0xMi13ZWVrLWFib3J0aW9uLWJhbi0yMDIzLTA1LTA0L9IBAA?oc%3D5&gl=FR&hl=en-US&cm=2&pc=n&src=1 ## 3 https://consent.google.com/ml?continue=https://news.google.com/rss/articles/CBMiWGh0dHBzOi8vd3d3LnJldXRlcnMuY29tL3dvcmxkL3VzL3NvdXRoLWNhcm9saW5hLWFkdmFuY2VzLTYtd2Vlay1hYm9ydGlvbi1iYW4tMjAyMy0wNS0xOC_SAQA?oc%3D5&gl=FR&hl=en-US&cm=2&pc=n&src=1 ## 4 https://consent.google.com/ml?continue=https://news.google.com/rss/articles/CBMib2h0dHBzOi8vd3d3LnJldXRlcnMuY29tL3dvcmxkL3VzL25vcnRoLWNhcm9saW5hLWxhd21ha2Vycy12b3RlLW92ZXJyaWRpbmctdmV0by0xMi13ZWVrLWFib3J0aW9uLWJhbi0yMDIzLTA1LTE2L9IBAA?oc%3D5&gl=FR&hl=en-US&cm=2&pc=n&src=1 ## 5 https://news.yahoo.com/nikki-haley-rallies-voters-south-182044998.html ## 6 https://www.theguardian.com/us-news/2023/may/05/mark-robinson-republican-north-carolina-governor-race ## urlToImage ## 1 <NA> ## 2 <NA> ## 3 <NA> ## 4 <NA> ## 5 https://s.yimg.com/ny/api/res/1.2/ufe_mO9XIoTjGU5jXTlnrA--/YXBwaWQ9aGlnaGxhbmRlcjt3PTEyMDA7aD02NzU-/https://s.yimg.com/hd/cp-video-transcode/production/73858b63-4d25-3398-bf84-4cc9a9c3ffac/2023-05-05/18-26-33/4b4e7fef-5b51-5d96-b3b7-0b6b3164ab40/stream_1920x1080x0_v2_3_0.jpg ## 6 https://i.guim.co.uk/img/media/86e85df1f545b7876edbe0be263eadcd593509de/0_270_5400_3240/master/5400.jpg?width=1200&height=630&quality=85&auto=format&fit=crop&overlay-align=bottom%2Cleft&overlay-width=100p&overlay-base64=L2ltZy9zdGF0aWMvb3ZlcmxheXMvdGctZGVmYXVsdC5wbmc&enable=upscale&s=34a3b41d309fe1aa597c353daf3168b3 ## publishedAt ## 1 2023-05-14T00:55:56Z ## 2 2023-05-04T02:37:00Z ## 3 2023-05-18T02:34:00Z ## 4 2023-05-16T10:09:00Z ## 5 2023-05-05T18:20:44Z ## 6 2023-05-05T09:00:19Z ## content ## 1 We use cookies and data to<ul><li>Deliver and maintain Google services</li><li>Track outages and protect against spam, fraud, and abuse</li><li>Measure audience engagement and site statistics to undeâ\200¦ [+1131 chars] ## 2 We use cookies and data to<ul><li>Deliver and maintain Google services</li><li>Track outages and protect against spam, fraud, and abuse</li><li>Measure audience engagement and site statistics to undeâ\200¦ [+1131 chars] ## 3 We use cookies and data to<ul><li>Deliver and maintain Google services</li><li>Track outages and protect against spam, fraud, and abuse</li><li>Measure audience engagement and site statistics to undeâ\200¦ [+1131 chars] ## 4 We use cookies and data to<ul><li>Deliver and maintain Google services</li><li>Track outages and protect against spam, fraud, and abuse</li><li>Measure audience engagement and site statistics to undeâ\200¦ [+1131 chars] ## 5 Republican presidential candidate Nikki Haley rallied voters during a campaign stop in Greer, South Carolina. (May 5) ## 6 To Mark Robinson, gay and transgender people are filth, homosexuality is an abominable sin, and the transgender movement is demonic and full of the spirit of the antichrist.\r\nMuslim Americans, meanwhâ\200¦ [+12729 chars]