Twitter live data mining using Spark streaming and Scala.

Want to work and learn live streaming data processing? Easiest way to create a twitter developer app and follow below code to ingest and store data in your AWS S3 for further analysis and processing with tools like Amazon EMR or Machine learning projects.

For deeper concepts connect me in linked. You may also outsource tech screening interview process to me.

 * @author Gyanendra
 * @Date : 08/12/19

import org.apache.spark.SparkConf
import org.apache.spark.streaming.twitter.TwitterUtils
import org.apache.spark.streaming.{Seconds, StreamingContext}
import twitter4j.auth.OAuthAuthorization
import twitter4j.conf.ConfigurationBuilder

object TweeterStreamReaderApp {
  def main(args: Array[String]) {

    var twitterCredentials = new Array[String](4);
    twitterCredentials(0) = "gA7xFE3S1QfVTN55Uuzb";
    twitterCredentials(1) = "2te2Z1yFvynXcp06rc2j3zg38tNAa1zY29rOT3d5BFI";
    twitterCredentials(2) = "1063309360480-61DChczOivazJZTWodLfuRRW8gDNfJ";
    twitterCredentials(3) = "bFYPmpiWhFgOtdJGe95YyhOntxOQAmx0xEYtF";

    val appName = "TweeterStreamReader"
    val conf = new SparkConf()
    val ssc = new StreamingContext(conf, Seconds(5))
    val Array(consumerKey, consumerSecret, accessToken, accessTokenSecret) = twitterCredentials.take(4)
    val filters = args.takeRight(args.length - 4)
    val cb = new ConfigurationBuilder
    val auth = new OAuthAuthorization(
    val tweets = TwitterUtils.createStream(ssc, Some(auth), filters)
    val englishTweets = tweets.filter(_.getLang() == "en")

    // lets print all rdd. Further you can store this to S3
    englishTweets.foreachRDD { (rdd, time) =>

    def p(rdd: org.apache.spark.rdd.RDD[_]) = rdd.foreach(println)

Download this code from my repo

Alexa, build deploy my project !!

Alexa, build deploy my project !! Thinking possible ?? Yep, it is possible.
did you leave office, and forgot to trigger build ? Don’t worry your Alexa can do this for you even if you in mid of Bangalore traffic 🙂 or enjoying Friday eve hangout far far away from your Boss 🙂 !
Keep visiting here I am gonna post how you can trigger build, deploy through Alexa ( not all projects at present but later chances are very high that any project can be built like this).

Keep visiting ……. to see updated content.

Spring boot code for AWS RDS in elasticbeanstalk

Are you new to aws RDS,elasticbeanstalk world and searching for a ready to use code for RDS connection? Try out out this sample project which connects RDS (mysql) from a web application (Spring boot + web ) and list out table records. You can run this web application in your local computer or in elasticbeanstalk (deploy rdselasticbean-0.1.war file). To test this code for your own RDS schema you need to change connection endpoint in record fetcher class. You may clone code from my gitlab repo.
Like it, if you find it helpful. Comment/Contact me if you face any issue while running this app. It’s all pretty straightforward, and quite handy for many kinds of elasticbeanstalk and RDS needs.

package com.example.rdselasticbean.db.DBrecordFetcher
(please make sure your RDS security group has proper access permissions for external IPs ) 

PS: Project is CI/CD enabled in case if anyone wants to test this in integrated environment.

Data display in JSP injected from DB operation class
RDS (mysql) server event view on Read and Write