Saturday, May 04, 2013

Play python with twitter streaming

I was interested in how to do twitter streaming. I tried to find out on the internet and got many examples. I tested it with python code(tweepy). I got data in JSON format. I thought it's a good idea, if I was able insert this data into mongodb.
So. I created my application on twitter and tested some a little bit code.
[surachart@centos ~]$ ./mongo/bin/mongo
MongoDB shell version: 2.2.2
connecting to: test
>
> db.twittertest.find()
> db.twittertest.find()
Used some code on the internet and changed a bit to insert data into database.
[surachart@centos ~]$ cat ./test-tweeter.py
#!/usr/bin/python
import tweepy
import pymongo
import json

consumer_key = "..."
consumer_secret = "..."
access_token_key = "..."
access_token_secret = "..."


auth1 = tweepy.OAuthHandler(consumer_key, consumer_secret)
auth1.set_access_token(access_token_key, access_token_secret)

class StreamListener(tweepy.StreamListener):
    def on_status(self, tweet):
        print tweet.text

    def on_error(self, status_code):
        print 'Error: ' + repr(status_code)
        return False

    def on_data(self, data):
        print 'Ok! Inserting Data.'
        from pymongo import MongoClient
        client = MongoClient()
        client = MongoClient('localhost', 27017)
        db = client.test
        test_id = db.twittertest.insert(json.loads(data))

l = StreamListener()
streamer = tweepy.Stream(auth=auth1, listener=l)
streamer.filter(track=['oracle'])
I filtered by using word 'oracle'.
[surachart@centos ~]$ ./test-tweeter.py
Ok! Inserting Data.

[surachart@centos ~]$ ./mongo/bin/mongo
MongoDB shell version: 2.2.2
connecting to: test
>
> db.twittertest.find()
{ "_id" : ObjectId("51851511cdc8722152c28171"), "contributors" : null, "truncated" : false, "text" : "Job Opportunity for Oracle PL/SQL Developers with HTML/Web development experience for 6+ Months(Contract) for Las...: http://t.co/p6p8cfV6qF", "in_reply_to_status_id" : null, "id" : NumberLong("330684010961448962"), "favorite_count" : 0, "source" : "<a href=\"http://www.linkedin.com/\" rel=\"nofollow\">LinkedIn</a>", "retweeted" : false, "coordinates" : null, "entities" : { "symbols" : [ ], "user_mentions" : [ ], "hashtags" : [ ], "urls" : [     {       "url" : "http://t.co/p6p8cfV6qF",       "indices" : [   118,    140 ],  "expanded_url" : "http://lnkd.in/G-hhiX",       "display_url" : "lnkd.in/G-hhiX" } ] }, "in_reply_to_screen_name" : null, "id_str" : "330684010961448962", "retweet_count" : 0, "in_reply_to_user_id" : null, "favorited" : false, "user" : { "follow_request_sent" : null, "profile_use_background_image" : true, "default_profile_image" : true, "id" : 360409067, "verified" : false, "profile_image_url_https" : "https://si0.twimg.com/sticky/default_profile_images/default_profile_4_normal.png", "profile_sidebar_fill_color" : "DDEEF6", "profile_text_color" : "333333", "followers_count" : 22, "profile_sidebar_border_color" : "C0DEED", "id_str" : "360409067", "profile_background_color" : "C0DEED", "listed_count" : 1, "profile_background_image_url_https" : "https://si0.twimg.com/images/themes/theme1/bg.png", "utc_offset" : null, "statuses_count" : 1777, "description" : null, "friends_count" : 0, "location" : "", "profile_link_color" : "0084B4", "profile_image_url" : "http://a0.twimg.com/sticky/default_profile_images/default_profile_4_normal.png", "following" : null, "geo_enabled" : false, "profile_background_image_url" : "http://a0.twimg.com/images/themes/theme1/bg.png", "name" : "Anand P. Sharma", "lang" : "en", "profile_background_tile" : false, "favourites_count" : 0, "screen_name" : "anandph123", "notifications" : null, "url" : null, "created_at" : "Tue Aug 23 04:52:03 +0000 2011", "contributors_enabled" : false, "time_zone" : null, "protected" : false, "default_profile" : true, "is_translator" : false }, "geo" : null, "in_reply_to_user_id_str" : null, "possibly_sensitive" : false, "lang" : "en", "created_at" : "Sat May 04 14:03:06 +0000 2013", "filter_level" : "medium", "in_reply_to_status_id_str" : null, "place" : null }
>
I was able to insert data...  and I will play with it later.


Monday, April 29, 2013

Interactive Data Visualization for the Web An Introduction to Designing with D3 By Scott Murray

Interactive Data Visualization for the Web An Introduction to Designing with D3 By Scott Murray.
This book was written about programming data visualization for non-programmer. You're able read this book and use it, even through you have no experience working with data or code.
This book will introduce you to D3, a JavaScript-based tool. You will be able to learn with many examples. Anyway, you might be able to begin with a series of tutorials. 

Benefits in this book:
  • Learn HTML, CSS, JavaScript, and SVG basics
  • Dynamically generate web page elements from your data—and choose visual encoding rules to style them
  • Create bar charts, scatter plots, pie charts, stacked bar charts, and force-directed layouts
  • Use smooth, animated transitions to show changes in your data
  • Introduce interactivity to help users explore data through different views
  • Create customized geographic maps with data
  • Explore hands-on with downloadable code and over 100 examples
When I began to read this book. I thought it might be difficult for me. I've no experience with data visualization, but not at all. I could understand and follow examples with my a little skill about java script. Anyway, I used http://alignedleft.com/tutorials/d3/ with my learning :)

I believe this book will give benefit for someone who is interested in interactive Data Visualization by using D3 and readers will enjoy with this book and examples
BTW, use https://github.com/alignedleft/d3-book/ to download codes.
Check book sampler.


Tuesday, March 26, 2013

MongoDB Applied Design Patterns Practical Use Cases with the Leading NoSQL Database By Rick Copeland


MongoDB is document-oriented database management system and supported by 10gen. Anyway, I was writing about a book, that's MongoDB Applied Design Patterns Practical Use Cases with the Leading NoSQL Database By Rick Copeland.

A book will guide you how to do the connection between MongoDB and the business problem it's designed to solve. Learn how to apply MongoDB design patterns with some Cases such as e-commerce, content management ans etc. You will see demonstrates from this book, that's very useful for readers. I believe this book is useful for someone who are interested in learning practical patterns for solving problems and designing applications using MongoDB.

This book has 2 parts: Part I introduces the reader to some generally applicable design patterns in MongoDB and Part II: Use Cases. This is good part. It covers a particular problem domain and the techniques and patterns used to address the problem:
  • Operational intelligence: Perform real-time analytics of business data
  • Ecommerce: Use MongoDB as a product catalog master or inventory management system
  • Content management: Learn methods for storing content nodes, binary assets, and discussions
  • Online advertising networks: Apply techniques for frequency capping ad impressions, and keyword targeting and bidding
  • Social networking: Learn how to store a complex social graph, modeled after Google+
  • Online gaming: Provide concurrent access to character and world data for a multiplayer role-playing game