0 - Hello and Welcome

-

Welcome!

Objectives of the blog as a whole:

  • Learn more about the open source tool Apache NiFi 
  • Use NiFi to query a Twitter feed for data
  • Use NiFi to push that data to an endpoint for public display
  • Gain knowledge on data science and data engineering

Style of the blog:

If you're following along with this blog, I hope that you will try to do the things you see in the posts.  With that said, I will do my best to explain how everything is done and make it as easy as possible for you, the reader, to recreate.  If you ever have any questions or comments along the way don't be afraid to leave a comment.

One thing I have found when looking at forums/tutorials/blogs is when the author includes bits that don't actually work, or intentionally adds a mistake- I will not do this.  I will also do my best to compartmentalize and label the parts of a given solution.  Hopefully this blog will serve as a useful resource to you.

The tool we will be using is called Apache NiFi and while there may be better solutions for the problems we will be tackling, there is little documentation on the web for developers of Apache NiFi.  With any luck this blog will become a useful resource for future users of the tool.

About me:

I am a senior at John Carroll University, which is located in Cleveland,OH.  I worked with Apache NiFi first at a company called TeleTracking Technologies Inc. over the summer of 2017 as an intern on their Analytics team.  Over the course of the summer myself and my intern partner became the "experts" of NiFi for the team... although we barely even scratched the surface of NiFi's potential.

Outside of computer science the two things I enjoy most are movies and soccer.  My love of film will most likely not be a part of this blog, but my love of soccer will.  In England there is a soccer team called AFC Wimbledon of which, for several reasons, I am a fan.  Their games are not broadcast in the U.S. and barely receive coverage in England as they are a "lower league" team.  That said, they still live tweet the games, so the goal of the project in this blog is to create an efficient way of keeping track of the games without sifting through a busy twitter feed.

Next time:

We will explore the basics of Apache NiFi by starting an instance on our computers.

Wanna get ahead?  You can find information about NiFi and the download we'll be using here:


Take it easy, Dude.


Comments

  1. Bernard EvenhuisNovember 21, 2017 at 8:39 AM

    I like your writing style and look forward to see this blog grow.

    ReplyDelete

Post a Comment

Popular posts from this blog

1 - Getting Started with Apache NiFi

-
Image
Getting Started with Apache NiFi Objectives: Learn about history of NiFi Install NiFi Start up NiFi for first time on local machine NiFi Overview: The project was created by the United States' infamous National Security Agency (NSA) and had the codename "Niagarafiles".  In 2014 the NSA released it as open-source software.  The primary developer of the project so far has been Hortonworks who are also well known for Hadoop and their suite of data science software. At its core, Apache NiFi is used to manage data flow.  This means that its job is to take in data, process it, then send it along to another destination.  Think of it as a mail service for data.  One of the advantages that it has is that uses a web based interface that is simple and allows users to manage the data flow in real time. Installing NiFi: I am installing NiFi on my Windows 7 PC, installing it on a Mac or Linux OS will follow a different procedure.  Also of note: We w...
Read more

2 - NiFi: UI, Processors, and Flow

-
Image
NiFi: UI, Processors, and Flow Objectives: NiFi Web UI Overview Create a processor and flow Start a flow NiFi Web UI Overview: First things first, fire up your local NiFi instance by running the run-nifi.bat file in the bin folder.  If you are new here, starting up the instance is detailed in the previous blog. Open up a web browser and access the NiFi UI on your local machine by going to this url: http://localhost:8080/nifi You should see your empty NiFi flow which is in the root Process Group.  A NiFi Flow is the whitish grid area.  This first flow is in the root Process Group, meaning everything is contained within this initial root flow.  Process Groups can be nested within each other, so we can make Process Groups within the root Process Group... Your screen should look like this: Blank NiFi UI On the top dark blue bar, you have several buttons that can be used to drag various items into your NiFi Flow.  Those items include: ...
Read more

4- Moving Data Out of NiFi

-
Image
4- Moving Data Out of NiFi Objectives: Examine data in Queues from GetTwitter Create flow from GetTwitter to Local Folder Use Templates to Create Multiple Flows Examine data in Queues from GetTwitter Last time we successfully pulled data from a Twitter account into our local instance of NiFi.  We are going to pick up right where we left off, so the NiFi Flow should look something like this (I have removed the GenerateFlowFile to Log Processors): So if you run the GetTwitter Processor without turning on the LogAttribute Processor, you will see the Queue in the link between the processors fill up.  You'll notice in the above picture that I have 976 files queued. Using these Queues is a good way to look at the data before it reaches a processor, which will be useful when debugging a problem when moving files through multiple processors in a larger flow. To see a file in the Queue, right click on the Queue, and click on the "List queue" option.  You will...
Read more