Wednesday, December 14, 2016

Connecting to hbase using Python

Connecting hbase by importing a package:

Source: https://media.readthedocs.org/pdf/happybase/latest/happybase.pdf ---- it is beautiful document that talks about hbase connection using python.
Source:#http://stackoverflow.com/questions/15808717/connecting-standalone-hbase-from-python
Script:

import happybase
connection = happybase.Connection('localhost', autoconnect=False)
connection.open()

Source: http://stackoverflow.com/questions/28456422/writing-to-hbase-table-from-python-happybase?rq=1

Connecting hbase using subprocess methods and extract data using get_value():

Status Success
Date: 11/3/2017
Script:

import csv
import json
import subprocess
import re
import pycurl
import StringIO
p1 = subprocess.Popen(["echo", 'get "%s","%s"' % (<table_name>,<unique_key>)], stdout=subprocess.PIPE)
p2 = subprocess.Popen(["hbase", "shell"], stdin=p1.stdout, stdout=subprocess.PIPE)
op_str=p2.communicate()[0]
column_value=get_value(op_str, <column_family>, <column_name>)
print column_value

Tuesday, December 13, 2016

Uses of Learning Python

Why should you learn Python Programming Language?

Source:http://stackoverflow.com/questions/1909512/what-is-python-used-for

Python offers a stepping stone into the world of programming. Even though Python Programming Language has been around for 25 years, it is still rising in popularity. Some of the biggest advantage of Python are it's

Easy to Read & Easy to Learn

Very productive or small as well as big projects

Big libraries for many things

What is Python Programming Language used for?

Source:http://stackoverflow.com/questions/1909512/what-is-python-used-for

As a general purpose programming language, Python can be used for multiple things. Python can be easily used for small, large, online and offline projects. The best options for utilizing Python are web development, simple scripting and data analysis. Below are a few examples of what Python will let you do:

Web Development:

You can use Python to create web applications on many levels of complexity. There are many excellent Python web frameworks including, Pyramid, Django and Flask, to name a few.

Data Analysis:

Python is the leading language of choice for many data scientists. Python has grown in popularity, within this field, due to its excellent libraries including; NumPy and Pandas and its superb libraries for data visualisation like Matplotlib and Seaborn.

Machine Learning:

What if you could predict customer satisfaction or analyse what factors will affect household pricing or to predict stocks over the next few days, based on previous years data? There are many wonderful libraries implementing machine learning algorithms such as Scikit-Learn, NLTK and TensorFlow.

Computer Vision:

You can do many interesting things such as Face detection, Color detection while using Opencv and Python.

Internet Of Things With Raspberry Pi:

Raspberry Pi is a very tiny and affordable computer which was developed for education and has gained enormous popularity among hobbyists with do-it-yourself hardware and automation. You can even build a robot and automate your entire home. Raspberry Pi can be used as the brain for your robot in order to perform various actions and/or react to the environment. The coding on a Raspberry Pi can be performed using Python. The Possibilities are endless!

Game Development:

Create a video game using module Pygame. Basically, you use Python to write the logic of the game. PyGame applications can run on Android devices.

Web Scraping:

If you need to grab data from a website but the site does not have an API to expose data, use Python to scraping data.

Writing Scripts:

If you're doing something manually and want to automate repetitive stuff, such as emails, it's not difficult to automate once you know the basics of this language.

Browser Automation:

Perform some neat things such as opening a browser and posting a Facebook status, you can do it with Selenium with Python.

GUI Development:

Build a GUI application (desktop app) using Python modules Tkinter, PyQt to support it.

Rapid Prototyping:

Python has libraries for just about everything. Use it to quickly built a (lower-performance, often less powerful) prototype. Python is also great for validating ideas or products for established companies and start-ups alike.

Python can be used in so many different projects. If you're a programmer looking for a new language, you want one that is growing in popularity. As a newcomer to programming, Python is the perfect choice for learning quickly and easily.

ANDDDD...

Python is a dynamic, strongly typed, object oriented, multipurpose programming language, designed to be quick (to learn, to use, and to understand), and to enforce a clean and uniform syntax.

Python is dynamically typed: it means that you don't declare a type (e.g. 'integer') for a variable name, and then assign something of that type (and only that type). Instead, you have variable names, and you bind them to entities whose type stays with the entity itself. `a = 5`makes the variable name `a` to refer to the integer 5. Later, `a = "hello"` makes the variable name `a` to refer to a string containing "hello". Static typed languages would have you declare `int a` and then `a = 5`, but assigning `a = "hello"` would have been a compile time error. On one hand, this makes everything more unpredictable (you don't know what `a` refers to). On the other hand, it makes very easy to achieve some results a static typed languages makes very difficult.

Python is strongly typed. It means that if `a = "5"` (the string whose value is '5') will remain a string, and never coerced to a number if the context requires so. Every type conversion in python must be done explicitly. This is different from, for example, Perl or Javascript, where you have weak typing, and can write things like `"hello" + 5` to get `"hello5"`.

Python is object oriented, with class-based inheritance. Everything is an object (including classes, functions, modules, etc), in the sense that they can be passed around as arguments, have methods and attributes, and so on.

Python is multipurpose: it is not specialised to a specific target of users (like R for statistics, or PHP for web programming). It is extended through modules and libraries, that hook very easily into the C programming language.

Python enforces correct indentation of the code by making the indentation part of the syntax. There are no control braces in Python. Blocks of code are identified by the level of indentation. Although a big turn off for many programmers not used to this, it is precious as it gives a very uniform style and results in code that is visually pleasant to read.

The code is compiled into byte code and then executed in a virtual machine. This means that precompiled code is portable between platforms.

Python can be used for any programming task, from GUI programming to web programming with everything else in between. It's quite efficient, as much of its activity is done at the C level. Python is just a layer on top of C. There are libraries for everything you can think of: game programming and openGL, GUI interfaces, web frameworks, semantic web, scientific computing...

Reading CSV file

Scenarios of CSV file:

Source: https://newcircle.com/s/post/1572/python_for_beginners_reading_and_manipulating_csv_files#opening-a-csv-file
These links have complete details of how to access csv file and work on it.

https://docs.python.org/2/library/csv.html ---- has complete details of read, write and more functions, with examples

Wednesday, December 7, 2016

For loop

Source: https://wiki.python.org/moin/ForLoop

Errors encountered while running scripts:

Indentation error:

Python follows a specific structure, like if loop is created inside for loop, there needs to be some space before you start if loop. Like the below, to justify the amount of space that needs to be given try to use compiler software's.
Script:

for key in list:
<------>if key == "name":
<----------->print "key is matching"

ImportError: No module named 'urllib2'

Source: http://stackoverflow.com/questions/2792650/python3-error-import-error-no-module-name-urllib2?noredirect=1&lq=1
Solution: In python 3 urllib2 was merged into urllib. See also another Stack Overflow question and the urllib PEP 3108.

Script: To make Python 2 code work in Python 3:

try:
import urllib.request as urllib2
except ImportError:
import urllib2

TypeError:

Script: urllib.request(jsonurl, jsonapi, headers)
TypeError: 'module' object is not callable

CSV error:

Error script: csv.Error: iterator should return strings, not bytes.
Soution Source: http://stackoverflow.com/questions/8515053/csv-error-iterator-should-return-strings-not-bytes

I just fixed this problem with my code. The reason it is throwing that exception is because you have the argument rb. Change that to r.
Script with issue:

import csv
ifile = open('sample.csv', "rb")
read = csv.reader(ifile)
for row in read :
print (row)

Script with solution:

import csv
ifile = open('sample.csv', "r")
read = csv.reader(ifile)
for row in read :
print (row)

ImportError: No module named happybase

NameError: name 'printfun' is not defined

Script that has issue:

printfun('123')
def printfun(msg):
print msg

O/P of the script:

Traceback (most recent call last):
File "Connectingtohbase.py", line 17, in <module>
printfun('123')
NameError: name 'printfun' is not defined

Resolution: <worked, as I have defined methods first(in the beginning) and accessed later>The solution to this problem is to invoke your classes and functions after you define them. Python does not have any way to forward declare classes or methods so the only option is to put the invocations of functions at the end of the program rather than the beginning. The other option is to put your methods in imported libraries at the top of your file which always get called first. – Eric Leschinski .
Solution script:

def printfun(msg):
print msg
printfun('123')

python JSON object must be str, bytes or bytearray, not 'dict:

Error occurred while reading json response
Command used: json.loads(<response_var>)
Fixed by using below:

v1= json.dumps(<response_var>)
json.loads(v1)

Justification:

json.loads take a string as input and returns a dictionary as output.
json.dumps take a dictionary as input and returns a string as output.

Source: http://stackoverflow.com/questions/42354001/python-json-object-must-be-str-bytes-or-bytearray-not-dict

Tuesday, December 6, 2016

Know Python (Mostly Syntax)

Define function:

Syntax:

def <function_name>(<values_of_parameters>):
<statements>
return <values>

Print:

Script:

print "\n\n======================================================================================================================================================"

print is inbuilt method that accepts string or variable, or both sometimes.
print msg_text //The simplest way to produce output is using the print statement where you can pass zero or more expressions separated by commas.

try: // colon(: ) is used after many inbuilt keywords like try, finally, for loop,if, predefined functions….
for row in request_data: // for loop syntax – keywords include ‘for’, ‘in’ and ‘:’
request_data = csv.reader(req_file) // csv was the inbuilt function imported and reader is the method
req_file = open('UK_Line_Inp.csv','rb') // Nothing related to open is imported not sure how open is working
import urllib, urllib2 // Open arbitrary resources by URL and extensible library for opening URLs
req = urllib2.Request(url, req_data, headers) // Same as above
rsp_response_code = response['response']['responseCode'] // ‘response’ is the object of load function. Here it means we are reading the value each element in json.
rsp_optimal_line = response['response']['line']['optimalLine'] // same as above
results_file_writer.writerow((row[0],row[1],row[2],row[3],rsp_optimal_line,'Pass' if row[3] == rsp_optimal_line else
except HTTPError as e:
except Exception as e:
except URLError as e:
results_file = open('UK_Line_API_Test_Results_'+timestamp +'.csv','wt') //what is ‘wt’ mode?

file object = open(file_name [, access_mode][, buffering]) //without file keyword code is being written.

- results_file_writer.writerow((row[0],row[1],row[2],row[3],'Error Response','Fail',req_data,pretty_resp))
- from urllib2 import Request, urlopen, URLError, HTTPError //To use the module bar, we can import it in two ways:

1.  import foo.bar

2.  from foo import bar

In the first method, we must use the foo prefix whenever we access the module bar. In the second method, we don't, because we import the module to our module's namespace.

- headers = {

'Content-Type':'application/json',

'client_app_id':'PZN',

'request_timestamp':'2016-09-06 13:53:59.030',

'channel':'WEB',

'request_id':'9999999802',

'country_code':'GB',

'client_platform':'ORCA'

}

- import subprocess
- Python is case sensitive (P and p are not same here) - “”(double quotes) and ‘’(single quote) are read same here

Run Python in windows

Steps:

Install Python on your machine. If it is already installed you could see "Python 3.6" folder in all programs section.
Go to command prompt, and then go the folder where python is installed. Here you enter "python", then you will see python version displayed on command prompt, if software is not installed you would see 'python is not recognized as an internal or external command'
From command prompt you could run python scripts. Below is the approach:
c:.....Programs\Python\Python36> python <filename.py> ----- this will run the python file.

Syntax to run the python script is "python <filename.py>

Source: Some of the information came from "Python Programming for Beginners: An Introduc…(Kindle Edition)"

Handson

12/6/16:

1. Reading json response:

import json

import urllib, urllib2

response = '{"status":"Success", "cust_data": [{"cat": "c1", "variables":{"name": "t1","value": "0" }}]}'

# Your code goes here

pretty_resp = json.loads(response)

print "Print json data: " ,pretty_resp

varval=pretty_resp["status"]

print "Valu of the variable: ", varval

O/P:

Print json data: {u'status': u'Success', u'cust_data': [{u'variables': {u'name': u't1', u'value': u'0'}, u'cat': u'c1'}]}

Valu of the variable: Success

2. Reading json from an array

import json

import urllib, urllib2

response = '{"status":"Success", "cust_data": [{"cat": "c1", "variables":{"name": "t1","value": "0" }}]}'

# Your code goes here

pretty_resp = json.loads(response)

print "Print json data: " ,pretty_resp

varval=pretty_resp["status"]["cust_data"]

print "Valu of the variable: ", varval

O/P:

Traceback (most recent call last):

File "prog.py", line 8, in <module>

TypeError: string indices must be integers

<Resolution> the above error could be removed by adding [0] after cust_data, as cust_data holds data in [].

3. Iterating through key and values:

a. import json

import urllib, urllib2

response = '{"status":"Success", "cust_data": [{"cat": "c1", "variables":{"name": "t1","value": "0" }}]}'

# Your code goes here

pretty_resp = json.loads(response)

print "Print json data: " ,pretty_resp

for key, value in pretty_resp.iteritems():

print key

O/P:

Print json data: {u'status': u'Success', u'cust_data': [{u'variables': {u'name': u't1', u'value': u'0'}, u'cat': u'c1'}]}

status

cust_data

b. import json

import urllib, urllib2

response = '{"status":"Success", "cust_data": [{"cat": "c1", "variables":{"name": "t1","value": "0" }}]}'

# Your code goes here

pretty_resp = json.loads(response)

print "Print json data: " ,pretty_resp

cust_datavals= pretty_resp["cust_data"][0]["variables"]

for key, value in cust_datavals.iteritems():

print key

O/P:

Print json data: {u'status': u'Success', u'cust_data': [{u'variables': {u'name': u't1', u'value': u'0'}, u'cat': u'c1'}]}

name

value

4. Extracting data from the [] and {} in json … I think indexes [0],[1]….while extracting data

from []

http://stackoverflow.com/questions/25105615/typeerror-list-indices-must-be-integers-not-str-while-parsing-json?rq=1

import json

import urllib, urllib2

response = '{"status":"Success", "cust_data": [{"cat": "c1", "variables":{"name": "t1","value": "0" }}]}'

# Your code goes here

pretty_resp = json.loads(response)

print "Print json data: " ,pretty_resp

#for key, value in pretty_resp.iteritems():

# print key

varval=pretty_resp["cust_data"][0]["variables"]["name"]

print "Value of the variable: ", varval

O/P:

Print json data: {u'status': u'Success', u'cust_data': [{u'variables': {u'name': u't1', u'value': u'0'}, u'cat': u'c1'}]}

Value of the variable: t1

Note: l['type'] gives you a list, not an object. So you have to access it like a list

        l['type'][0]["references"]

Note: Key could be variable name as aswell like this:

custdata="cust_data"

cust_datavals= pretty_resp[custdata][0]["variables"

# Validate values of the variabls in API response. But no validation is done on category and sub category.
import json
import urllib

response = '{"status":"Success", "cust_data": [{"cat": "c1", "variables":[{"name": "dfdf","value": "0" },{"name": "mo_snc","value": "10" }]},{"cat": "c2", "variables":[{"name": "trd_losed","value": "0" },{"name": "trd_lse_d","value": "10" }]}]}'# Your code goes herepretty_resp = json.loads(response)
print("Print json data:pretty_resp=  " ,pretty_resp)
#for key, value in pretty_resp.iteritems(): #   print key#i is number of cust data layersicount=len(pretty_resp["cust_data"])
print("Number of cust data layers: ",icount)

for i in range(0,icount):
     print("i= ",i)
     jcount = len(pretty_resp["cust_data"][i]["variables"])
     for j in range(0,jcount):
         print("j= ", j)
         varname=pretty_resp["cust_data"][i]["variables"][j]["name"]
         print("variable name: ",varname)
         if(varname=="mo_snc"):
             varvalue=pretty_resp["cust_data"][i]["variables"][j]["value"]
             if(varvalue=="10"):
                 print("Name: ",varname,"& value is: ",varvalue)
                 print("Script has passed")
             else:
                 print("Name: ",varname, "value is: ",varvalue,", so not matching")
             break     breakprint('Script has completed running')