6- Finale Part 1: NiFi

-

6 - Finale Part 1: NiFi

Objectives:

  • Current State of the Flow
  • Processing Data
  • Where's it all going?

Current State of the Flow

Before I get into this, I would like to briefly remind you what our goal is... Take data from Twitter and display it at some endpoint and try to make it look nice.

Spoiler Alert: It's not working out for us this time, but we're going to put together the pieces so that it can happen in the future.

Last post I said that we would be switching to using Tableau, and we're still going to try something with that... I'm not sure if it will work.  Now we are adding a step in the process: Google Cloud.

What will the Google Cloud allow us to do? Google Cloud Platform (GCP) uses these objects called "buckets" to store information in the cloud.  NiFi has a PutGCP processor, which we will utilize to connect NiFi to GCP.  From there we should be able to connect up to Tableau, but that will be for next time.

Here is a visual breakdown of the plan:

Processing Data

Here is our current flow with the PutGCP process in it.  We also have two GetTwitter Processors pointed at two soccer teams that I follow on Twitter.  I have queued up several thousand tweets.




In order to get PutGCP to work, you will need to set up a project on the Google Cloud Platform...

this can be started by going to cloud.google.com.

Because we are connecting to a resource outside of NiFi, we also need to set up a Controller Service.  This is fairly straightforward: Make sure you have the NiFi Flow selected in the Operate menu on the left hand side of the interface.  Then go into the configuration and click on the Controller Services Tab.

Once you add the PutGCP Processor, it will tell you what kind of Controller Service you need to create, and it can be enabled and configured (using the information from your GCP console) in this tab of the NiFi Flow Configuration.

So where does it go?

NiFi can only send it out into GCP buckets, which are just cloud storage containers that can be accessed by other Google programs.  They can also be accessed by Tableau, which is what we will be using it for.  When you run the flow, if you have set it up correctly, you will see JSON files appearing in the assigned GCP bucket


Next Time:

The Grand Finale of this stretch of blog posts will conclude in an investigation of Tableau software for visualizing 
Thanks for reading this blog post!

Take it easy, Dude.

Comments

Post a Comment

Popular posts from this blog

1 - Getting Started with Apache NiFi

-
Image
Getting Started with Apache NiFi Objectives: Learn about history of NiFi Install NiFi Start up NiFi for first time on local machine NiFi Overview: The project was created by the United States' infamous National Security Agency (NSA) and had the codename "Niagarafiles".  In 2014 the NSA released it as open-source software.  The primary developer of the project so far has been Hortonworks who are also well known for Hadoop and their suite of data science software. At its core, Apache NiFi is used to manage data flow.  This means that its job is to take in data, process it, then send it along to another destination.  Think of it as a mail service for data.  One of the advantages that it has is that uses a web based interface that is simple and allows users to manage the data flow in real time. Installing NiFi: I am installing NiFi on my Windows 7 PC, installing it on a Mac or Linux OS will follow a different procedure.  Also of note: We w...
Read more

2 - NiFi: UI, Processors, and Flow

-
Image
NiFi: UI, Processors, and Flow Objectives: NiFi Web UI Overview Create a processor and flow Start a flow NiFi Web UI Overview: First things first, fire up your local NiFi instance by running the run-nifi.bat file in the bin folder.  If you are new here, starting up the instance is detailed in the previous blog. Open up a web browser and access the NiFi UI on your local machine by going to this url: http://localhost:8080/nifi You should see your empty NiFi flow which is in the root Process Group.  A NiFi Flow is the whitish grid area.  This first flow is in the root Process Group, meaning everything is contained within this initial root flow.  Process Groups can be nested within each other, so we can make Process Groups within the root Process Group... Your screen should look like this: Blank NiFi UI On the top dark blue bar, you have several buttons that can be used to drag various items into your NiFi Flow.  Those items include: ...
Read more

4- Moving Data Out of NiFi

-
Image
4- Moving Data Out of NiFi Objectives: Examine data in Queues from GetTwitter Create flow from GetTwitter to Local Folder Use Templates to Create Multiple Flows Examine data in Queues from GetTwitter Last time we successfully pulled data from a Twitter account into our local instance of NiFi.  We are going to pick up right where we left off, so the NiFi Flow should look something like this (I have removed the GenerateFlowFile to Log Processors): So if you run the GetTwitter Processor without turning on the LogAttribute Processor, you will see the Queue in the link between the processors fill up.  You'll notice in the above picture that I have 976 files queued. Using these Queues is a good way to look at the data before it reaches a processor, which will be useful when debugging a problem when moving files through multiple processors in a larger flow. To see a file in the Queue, right click on the Queue, and click on the "List queue" option.  You will...
Read more