3 - Getting Twitter To Talk to NiFi using Get Twitter

Getting Twitter to Talk to NiFi using Get Twitter

Objectives:

• Create a GetTwitter Processor in NiFi
• Set up Twitter to talk to NiFi
• Create a flow using GetTwitter

Create a GetTwitter Processor in NiFi

Welcome back! Today we are going to use NiFi's GetTwitter processor to create a flow of data into NiFi.  To start, we need to start our local NiFi instance and add a GetTwitter processor.  Your flow should look something like this:

You'll notice two things - First that our GenerateFlowFile --> LogAttribute Flow is still there from the last time, which is good to see. Second - GetTwitter isn't linked to anything and won't be able to do anything until it's configured.

Let's open the configuration, and go to the Properties Tab to see what we need:

The first entry determines what type of Endpoint we are creating.  This will essentially determine if we want to see all Tweets in real time or if we want to pick which Twitter feeds to follow.  Pick the Filter Endpoint option so that we can control which feeds it pulls data from.

The Consumer Key, Consumer Secret, Access Token, and Access Token Secret all come from Twitter.  In order to create these values, we need to create a Twitter app.  

Set up Twitter to talk to NiFi

You must have a Twitter account to do this, and the navigate to apps.twitter.com and click Create New Application, it will take you here... I have filled in the details for my application, please consider trying to make one for you:

For now I entered my stuweb website.  This information can be changed later.  The application I want to make takes in Twitter feeds from my favorite soccer teams.  For right now, I will target a small team called AFC Wimbledon.

Once you create an Application, you will be taken to the Application Management screen.  Go to Keys and Access Tokens.  Here you will be able to generate the Consumer Key, Consumer Secret, Access Token, and Access Token Secret for the Properties in NiFi.  Below I have the test information that I used and it is no longer relevant so it is okay to be shared, but in general you should keep the Secret values hidden from human view.

Create a flow using GetTwitter

So now we will fill in the details for consumer and token, then we want to Filter to a specific account, so we need to put the ID of the account  in IDs to Follow.  My research into this revealed that each Twitter user has an 8 digit ID and this is what NiFi is looking for.  The best way to get this ID is a website called: gettwitterid.com

Surprisingly this worked, so I got the ID for AFCWimbledon and entered it into the field.  I encourage you to try this out with your own twitter feed, you can also add multiple IDs by putting commas between them, this will become more useful later on.  Here is our final configuration:

Without any warnings, you should be able to run the flow.  In order to verify that it is working, point the GetTwitter processor to a LogAttribute Processor.  When you run the flow for a little bit and refresh, you will see the Queue fill up.

You're probably thinking: "Well that's great, but what the hell is in that queue?"

Try figuring it out for yourself and next time we'll explore the answer to that question and more!

Next Time:

Thanks for reading this post!

We're starting to get into the meat and potatoes and the "fun stuff" that makes me get up and cheer every time something works.  Next time we'll be looking at the data we get from the Twitter processor and seeing if we can save it to our local machines.

Hope to write for you again soon, but until then-

Take it easy, Dude.