Twitter is all about what is happening and what people are talking about right now. Existing analytics help unlock the business value of discussions on Twitter, but most of these tools don’t provide a live view about what’s happening or what actions you should take in real time. In today’s post, I’m excited to partner with Travis Collins, the founder of datapm, to share how Timeplus and DataPM can help you develop a real-time Twitter marketing app within a few minutes, without a single line of code. This makes it super easy to create real-time social listening apps to understand how customers are talking about your company or products, and plan your next social marketing campaign with trending hot topics.
Seeing is believing. You can check this short video:
Or you can try live demonstration systems:
You can send a new tweet with the #timeplus tag, such as “#timeplus is awesome!” then within 15 seconds, your tweet will be analyzed and you can see how others are thinking of the product.
How it works
At a high level, there are just three simple steps:
Load the real-time twitter data to Timeplus, via datapm command-line interface (CLI)
Build Streaming-SQL to understand the hot hashtags or how others think of your product/company
Visualize the real-time insights via charts/dashboards, or take real-time actions via calling Twitter API
Let’s dive in.
Introducing datapm
datapm (Data Package Manager) is an open source data exchange platform.
You can publish your own public or private packages and even host your own data registry for private B2B exchanges. Check out datapm.io for more information.
In our example, we will be using the command line interface of datapm, which is available for Windows/Linux/Mac, to easily load real-time tweets and send them to Timeplus.
Note: datapm command line is also a great tool to load other data to Timeplus, such as PostgreSQL, Redshift, local files, Google Sheets, and Bitcoin live prices.
Load real-time tweets to Timeplus via datapm
Twitter provides a real-time API to programmatically retrieve and analyze Twitter data. Please follow the docs to get access to the v2 Twitter API.
You will also need to download datapm and install it locally. https://datapm.io/downloads
In the recent versions of datapm, the Twitter source and Timeplus sink are added to make the integration extremely easy.
Make sure the datapm version is 0.31.23 or above.
datapm --version
0.31.23To load the real-time tweet, simply run
datapm fetch twitter/sampleThis will get roughly 1% random sample of publicly available Tweets in real-time.
Once you have the Twitter API token, you can load data via datapm with a single command:
datapm fetch twitter/sample
datapm client version 0.31.23 on darwin-x64
Source Selection
✔ Found package twitter/sample
ℹ 1 schemas with more than 2k records
tweets Schema Options
✔ Exclude any attributes from tweets? › No
✔ Rename attributes from tweets? › No
Inspecting twitter
Twitter Credentials
✔ Twitter API Bearer Token? … ******************************************************************************************************
✔ Authentication succeeded
ℹ Connecting to twitter.comThen you can send the real-time data to Timeplus. Simply join the Beta program of Timeplus Cloud and to receive an API token for your Beta tenant.
Choose Timeplus as the sink:
Sink Connector
? Sink Connector? ›
Big Query
Console (Standard Out)
Decodable
Kafka
Local File
MongoDB
MySQL
PostgreSQL
Redshift
❯ TimeplusThen set the hostname for the Timeplus beta tenant:
Timeplus Connection
? Repository? ›
❯ datapm.beta.timeplus.com
New RepositorySet the API token and type the stream name. If the data stream doesn’t exist, it will be created automatically with the proper data schema. In this example, let’s name the stream as “tweets”:
Timeplus Configuration
✔ Stream for tweets records? … tweets
✔ Created Timeplus Stream tweets
Checking State of tweetStream
✔ New records may be available
Reading tweetStream
✔ text has integer and string values. How should this conflict be handled? › Cast all values as string
⠏ Stream tweetStream
301 records received - 28.6 per secondThat’s it! Then go to the Timeplus Cloud web UI and click the ‘tweets’ stream. A SQL is generated:
SELECT * FROM tweets You will see the live tweets immediately.
To get the tweets relevant to your product or company, load data with `datapm fetch twitter/stream`:
datapm fetch twitter/stream
Source Selection
✔ Found package twitter/stream
ℹ 1 schemas with more than 124 records
tweets Schema Options
✔ Exclude any attributes from tweets? › No
✔ Rename attributes from tweets? › No
Inspecting twitter
Twitter Credentials
✔ Credentials? › AAAAA...AlIT3
✔ Authentication succeeded
ℹ Connecting to twitter.com
✔ Select Twitter Stream Type › Search/Filter Stream
✔ Twitter Search Query? … #timeplus
Sink Connector
✔ Sink Connector? › Timeplus
✔ Found the connector named Timeplus
Timeplus Connection
✔ Repository? › datapm.beta.timeplus.com
✔ Connection successful
Timeplus Credentials
✔ Credentials? › ***
✔ Authentication succeeded
Timeplus Configuration
✔ Stream for tweets records? … tweets_timeplus
✔ Created Timeplus Stream tweets_timeplus
Checking State of tweetStream
✔ New records may be available
Reading tweetStream
⠙ Stream tweetStream
1 records received - 0.0 per secondThen post a tweet with the specific keywords. Within 10-15 seconds, this tweet will be available in the search results:
Analyze real-time tweets with streaming SQL
Once the real-time data arrives in Timeplus, you can run all kinds of analysis to gain real-time insight. Please check our Top 10 Streaming SQL Patterns for details.
To understand the most popular hashtags in the real-time tweets, you can extract the hashtag from the tweets (if any). The Timeplus UI will show the top 3 tags automatically in the live view.
To determine the sentiment of the tweets (whether the tweets about your product are positive or negative), you can use online services or NLP (natural language processing) programming libraries to do the accurate analysis. Just for demonstration purposes, in this example, we can use SQL to do keyword-based analysis.
Now you have real-time insights. What’s next?
As we demonstrate in many other case studies, you can visualize the real-time insights by building a live dashboard with streamlit or other BI tools, or you can take real-time actions automatically via webhook or AWS Lambda. For example, you can create an “auto-reply” process, when anyone tweets about your product publicly on Twitter, you can add a reply to manage customers and create ServiceNow tickets or support tickets automatically to track the issue. Or even as simple as “like” and “retweet” the tweets if anyone shares a positive product review on twitter. This would greatly improve customer satisfaction and bring customers closer to your social media account and community.
Conclusion
At Timeplus, we are pushing the limits to build real-time applications much faster and much easier than ever before. We are excited to partner with DataPM to together deliver on this vision. Our blog shows that you don’t need coding to get real-time Twitter data into Timeplus and gain real-time insights. All you have to do is to join the Timeplus Cloud beta, download the datapm tool, and connect things together via API tokens. Fast, powerful, intuitive.
Join our Timeplus community or sign up for our private beta to learn more. We look forward to hearing your feedback!