Dataswap

Using APIs

Thom Neale:

• twitter: @twneale
• github: twneeale

Government Data

• Sunlight Foundation API portal
  • in python, "pip install sunlight" python-sunlight
  • Congress API
    • Look up legislators by zip code or lat/long
    • Official Twitter, YouTube, and Facebook accounts.
    • Committees and subcommittees in Congress, including memberships and rankings.
    • Visualizing partisanship
    • Cody's talk showed grouping of federal legislators by frequency of voting similarity
    • Do stuff like this: an area curve depicting the ideological distribution of the Pennsylvania house and senate
    • Useful libraries
      • Political Science Computational Library
      • Absurd amount of Political Science computation insanity
      • pypscl
      • A friendlier Python wrapper for said computational insanity
  • Influence Explorer API
    • See it in action: http://influenceexplorer.com/
    • Programmatic access tracking the influence of money in politics
    • Search and filter individual contributions, contracts, grants and lobbying registrations.
    • Find the most active Politicians, Individuals, Organizations and Industries based on receipts, expenditures and lobbying.
  • Openstates.org API
    • See it an action at http://openstates.org
    • State Metadata - Details on what data is available, including terms, sessions, and state-specific names for things.
    • Bills - Details on bills & resolutions, including actions & votes.
    • Legislators - Details on legislators, including contact details.
    • Committees - Details on committees as they currently stand.
    • Events - Details on upcoming events such as committee meetings and hearings.
    • Districts - Details on districts and their boundaries.
    • What kinds of stuff could you do with this data?
    • Predictive analytics for state legislations: FiscalNote.com
      • No one is doing this stuff (except fiscalnote.com)
      • Does anyone care? Mark Cuban just invested 1.2 Million in Fiscal Note
        
    • Terrible Ideas
    • Legislative Hot or Not .COM
    • Republican/Democrat or not .COM
• Canadian Legal Information Institute
  • Canadian Court Decisions API
  • Network analysis of Canadian Court decision citations patterns by me!
  • Cite Fight! cite-fight.com
• OpenCorporates.org API
  • How complex are financial relationships? (OMG)[http://opencorporates.com/viz/financial/]

Other useful APIs

• Diffbot.com
  • Extract main content from a page (remove navigation, header, footer, etc)
  • Extract keywords
• Alchemy API
  • Entity recognition--returns entities tagged with dbpedia, freebase IDs
  • Sentiment analysis
• SmartyStreets
  • Pass it unstructured text, get one or more normalized addresses in return
• Open Calais
  • Entity extraction

5 Tips for Working with APIs

• Be cheap
  • Resist the urge to subscribe to non-free APIs, at least at first
  • Too often developers overcommit themselves to a certain technology--don't do it!
  • Try out different free APIs; throw out the ones that don't work for you
• Later, inquire about free/cheap use for educational purposes
  • Some service will give you free unlimited use
  • Others only do that for 501(c)(3) nonprofit orgs
• Free APIs can be flaky
  • Consider using an task queue like Celery
  • Tasks succeeded or fail individually, can be retried individually
• To maximize freeness, consider caching responses
  • Many APIs other data in JSON
  • You can insert those responses directly into NoSQL databases like MongoDB, CouchDB, Cassandra
  • MongoLab You can create an unlimited (?) number of free 500MB of MongoDB databases
  • Resist the urge to cache response on disk: if you're using Amazon EC2, you'll be sorry

GO FORTH AND HACK!