Love of Python : My first Python script

Every next day my love of Python and Scala is getting stronger, Hence I thought to post one python script here which is a sample script to perform some common configurations that we need in application deployment on production, test or automation servers. I am going to use this script (updated version of it) in my project to reduce an overtake of approximately 15-20 mins of manual configuration works of service deployment.

This python script performs following tasks
(1) Add new configuration
(2) Update configurations
(3) Run Unix commands (native OS) on server and few other tasks

You can customize this script as per your project requirement. The script is available at my gitlab repository. I would love to improvise any of my reader’s recommendations for any specific change/requirement over this script.

PS: This script is in the initial draft form, I will update it with more useful commands and comments to make it more user-friendly.

 

 

Handling dynamic/unknown Datasource in Application

Let’s see how we can deal with a requirement when server-side input configuration or data source is not constant in your application and furthermore memory footprint of application should be also very low ( i.g a mobile Product ).

Once for a big data ( web crawler based ) product we had one requirement where input datasource was not fixed, based on customer choice datasource was configurable. Since product was a mobile application so we wanted to keep application’s memory footprint as low as possible, so we did not use Spring / Hibernate / Gauva etc…. framework for creating and maintaining configuration container on Fly.

Here is one basic design approach that I had prepared which later I improved to meet exact requirement. Contact me if you need further details on such implementations. I will upload this design in SkilledMind’s (http://skilledminds.in/) gitlab.

PS: Due to an agreement I have not added exact design.

Data Access layer Design

 

 

An Introduction to REST and a High Level Design of a REST API

What is REST and Why is it called Representational State Transfer?

REpresentational State Transfer (REST) is a style of software architecture for distributed systems such as the World Wide Web. REST has emerged as a predominant Web service design model.
The Web is comprised of resources. A resource is any item of interest. For example, the Jet Aircraft Corp may define a 747 resource. Clients may access that resource with this URL:
http://www.jet.com/aircraft/747
A representation of the resource is returned (e.g., Jet747.html). The representation places the client application in a state. The result of the client traversing a hyperlink in Jet747.html is another resource is accessed. The new representation places the client application into yet another state. Thus, the client application changes (transfers) state with each resource representation –> Representational State Transfer!

Below is one sample high level REST API design that I had designed some year back. REST’s new learner can take reference of it.

Sample search REST API High level design

Spark Exceptions

List of exceptions and troubleshooting steps that I have encountered till now using Spark in my project.

Issue 1 # Exception in thread “main” org.apache.spark.SparkException: A master URL must be set in your  configuration

Fix :   It means you simply forgot to specify the master URL.
SparkConf configuration = new SparkConf() .setAppName(“Your Application Name”) .setMaster(“local”);

Issue 2 # <Next_Exception_will_be_listed_soon>

Kafka Exceptions

Kafka is super buzzword nowadays in Bigdata space, so I thought to share some of exceptions and troubleshooting steps that I have encountered till now using Kafka in my project.

Issue 1 # Error while fetching metadata with correlation id 0 : {Visitor=LEADER_NOT_AVAILABLE (org.apache.kafka.clients.NetworkClient)

Fix :  There could be many reason of this failure but I resolved it by updating
                           host.name=localhost
                           advertised.host.name=localhost
        in $Kafka_home/config/server.properties where localhost is kafka’s server                   hostname. Basically it was happening due to incorrect network binding of my               laptop’s wireless interface.

Issue 2 # <Next_Exception_will_be_listed_soon>

What Is Middle Tier Clustering?

Middle tier clustering is just a cluster that is used for service the middle tier in a application. This is popular since many clients may be using middle tier and a lot of heavy load may also be served by middle tier that requires it be to highly available.

Failure of middle tier can cause multiple clients and systems to fail, therefore its one of the approaches to do clustering at the middle tier of a application.

In java world, it is really common to have EJB server clusters that are used by many clients. In general any application that has a business logic that can be shared across multiple client can use a middle tier cluster for high availability.

To reach Big Data take your first step with JSON

Ten years ago, XML was the primary data interchange format. When it came on the scene, it was a breath of fresh air and a vast improvement over the truly appalling SGML (Standard Generalized Markup Language).

But it’s no secret that in the last few years, a bold transformation has been afoot in the world of data interchange. The more lightweight, bandwidth-non-intensive JSON (JavaScript Object Notation) has emerged not just as an alternative to XML, but rather as a potential full-blown successor.The rise of JSON as a key player in database technologies is another bad portent for XML. As it stands, Big Data does not have a preferred data interchange format per se. But the claim that I’d like to make about Big Data and JSON is a bit more specific. What I’d like to argue is that JSON is emerging as a preferred format in web-centric, so-called “NoSQL” databases.

I am putting one JSON example here. you may try to write corresponding XML file and a java Parser for your learning.


{
  "Users": {
    "type": "bank",
    "User": [
      {
        "name": "Ram",
        "Bank": [
          {
            "name": "SBI",
            "UserID": "RamSBI",
            "Password": "RamSBIpwd"
          }
        ]
      },
      {
        "name": "Mohan",
        "Bank": [
          {
            "name": "SBI",
            "UserID": "MohanSBI",
            "Password": "MohanSBIpwd"
          }
        ]
      } 
    ]
  }
}