5- Major Brick Wall

-

5- Major Brick Wall

Objectives:

  • Address the Problem
  • Find a Solution
  • Begin Implementing Solution
In his talk, which has become known as The Last Lecture, Randy Pausch discusses the many challenges a person faces in their life as they attempt to achieve their goals.  He calls them "brick walls" and says that they are there to help you figure out how to get around them.  
We have hit a major brick wall.

I am intentionally making this post uninteresting so that you skip it and go to the next post, which will solve the problems outlined in this post.

Address the Problem

The next step in our NiFi flow is to process the data we are getting from GetTwitter.  We know that it is providing us with a JSON file, which is great, but it is raw data.  This JSON file, unprocessed, is practically useless to us.  It provides us with WAY too much information, and some of the files that we are outputting are not even real tweets.

Boiled down, the problem is that we have data, but it doesn't look nice.

Find a Solution

I have looked for several solutions to this problem.  First, I believed that it might be best try converting the JSON into something like a CSV so that it could be brought into R or parsed in a piece of software that we could write that parses "flat" text files.  This would work out great, if NiFi had a simple solution for converting JSON to CSV, which it currently does not.  My attempts to make this a reality using various processors fell flat on its face and was way too complex for the problem at hand.

Next, I looked into pushing the data out of NiFi in a format that JSON could convert to... like SQL.  While it's not native to NiFi, converting JSON to an SQL statement is something I have done in the past.  This task would require a target SQL database, which while possible to set up, would be time consuming and need to be explained on here.

As far as the front end of the data visualization would go, I dreamed of building a high quality UI for the task, but my interest in Tableau lead me to want explore this application as a tool for achieving that same goal.

Once opening Tableau and playing around for a little bit, I managed to connect it up to the folder that I pushed the Twitter Output to and oh my goodness look at that, no need for a database.  We will be using Tableau as the endpoint for our data.

Setting Up Tableau.

Tableau is a proprietary software, I am new to it, this will be a big learning experience for all... I will be using this for 14 days, starting today... I'm not even positive that this will work.
Please find the link to the download of Tableau here:
https://www.tableau.com/products/trial

When asked about a business, I put John Carroll University information into it.

In Tableau add a new JSON data source to your new project.  Choose a file in your folder with the Twitter Output... hopefully it looks like this:



So next time we will try to figure out what the hell Tableau does and how to use it for this project.

Next Time:

I'm sorry that I had to make this post... I hope you didn't make it this far and just moved onto the next post...

Thanks for reading this blog post!

Take it easy, Dude.

Comments

  1. UnknownNovember 20, 2017 at 8:06 AM

    This blog is really going downhill.

    ReplyDelete
  2. Bernard EvenhuisNovember 21, 2017 at 8:38 AM

    This comment has been removed by the author.

    ReplyDelete
  3. Bernard EvenhuisNovember 21, 2017 at 8:43 AM

    I appreciate your endeavor to use Tableau, which can often be difficult to work with, to visualize data. God speed!

    ReplyDelete

Post a Comment

Popular posts from this blog

1 - Getting Started with Apache NiFi

-
Image
Getting Started with Apache NiFi Objectives: Learn about history of NiFi Install NiFi Start up NiFi for first time on local machine NiFi Overview: The project was created by the United States' infamous National Security Agency (NSA) and had the codename "Niagarafiles".  In 2014 the NSA released it as open-source software.  The primary developer of the project so far has been Hortonworks who are also well known for Hadoop and their suite of data science software. At its core, Apache NiFi is used to manage data flow.  This means that its job is to take in data, process it, then send it along to another destination.  Think of it as a mail service for data.  One of the advantages that it has is that uses a web based interface that is simple and allows users to manage the data flow in real time. Installing NiFi: I am installing NiFi on my Windows 7 PC, installing it on a Mac or Linux OS will follow a different procedure.  Also of note: We w...
Read more

2 - NiFi: UI, Processors, and Flow

-
Image
NiFi: UI, Processors, and Flow Objectives: NiFi Web UI Overview Create a processor and flow Start a flow NiFi Web UI Overview: First things first, fire up your local NiFi instance by running the run-nifi.bat file in the bin folder.  If you are new here, starting up the instance is detailed in the previous blog. Open up a web browser and access the NiFi UI on your local machine by going to this url: http://localhost:8080/nifi You should see your empty NiFi flow which is in the root Process Group.  A NiFi Flow is the whitish grid area.  This first flow is in the root Process Group, meaning everything is contained within this initial root flow.  Process Groups can be nested within each other, so we can make Process Groups within the root Process Group... Your screen should look like this: Blank NiFi UI On the top dark blue bar, you have several buttons that can be used to drag various items into your NiFi Flow.  Those items include: ...
Read more

4- Moving Data Out of NiFi

-
Image
4- Moving Data Out of NiFi Objectives: Examine data in Queues from GetTwitter Create flow from GetTwitter to Local Folder Use Templates to Create Multiple Flows Examine data in Queues from GetTwitter Last time we successfully pulled data from a Twitter account into our local instance of NiFi.  We are going to pick up right where we left off, so the NiFi Flow should look something like this (I have removed the GenerateFlowFile to Log Processors): So if you run the GetTwitter Processor without turning on the LogAttribute Processor, you will see the Queue in the link between the processors fill up.  You'll notice in the above picture that I have 976 files queued. Using these Queues is a good way to look at the data before it reaches a processor, which will be useful when debugging a problem when moving files through multiple processors in a larger flow. To see a file in the Queue, right click on the Queue, and click on the "List queue" option.  You will...
Read more