To monitor our Internet bandwidth usage I wrote a Node.js program to scrape the data from the admin web UI for my Verizon Actiontec MI424WR router. Here‘s the code for that.

Last week I upgraded my internet to Verizon Fios Gigabit and with that upgrade, the Actiontec router was replaced with another router: a Netgear R7000 running Advanced Tomato (open source Linux-based firmware for Broadcom based Wi-fi routers). Advanced Tomato has a pretty click interface to monitor bandwidth, but I still want the data in my Splunk instance.

Luckily, Advanced Tomato runs a variant of Linux, so all I needed was a shell script to calculate bandwidth usage data and send to Splunk via the Splunk Http Event Collector.

I found a script by WaLLy3K that already had the bandwidth calculation logic and all I had to add was a little more code to send the data to Splunk.

Step-by-step Instructions

Enable JFFS Partition on Your Router

Enable JFFS Partition on your router so that you have permanent storage for your script. Otherwise if you saved your script in /tmp, it’ll be gone after the next reboot. Log into your router’s admin UI, choose Administration/JFFS, select Enabled and Save.

Create Your Script

SSH into your router and create a shell script at /jffs/bandwidth.sh with the content from here. Update the splunkUrl variable with your Splunk HEC URL. If you are not able to SSH, make sure you have SSH Daemon enabled under Administration/Admin Access.

For more info on installing Splunk HTTP Event Collection, see my previous post.

# this is just an excerpt of the code. For full code see 
# https://github.com/chinhdo/shell-scripts/blob/master/sh/bandwidth.sh

...
wan_iface=`nvram get wan_iface`
calc(){ awk "BEGIN { print $*}"; }    # Calculate floating point arithmetic using AWK instead of BC

checkWAN () {
    [ -z $1 ] && sec="1" || sec="$1"

    netdev=`grep "$wan_iface" /proc/net/dev`
    pRX=$(echo $netdev | cut -d' ' -f2)
    pTX=$(echo $netdev | cut -d' ' -f10)
    sleep $sec
    netdev=`grep "$wan_iface" /proc/net/dev`
    cRX=$(echo $netdev | cut -d' ' -f2)
    cTX=$(echo $netdev | cut -d' ' -f10)

    [ $cRX \< $pRX ] && getRX=`calc "$cRX + (0xFFFFFFFF - $pRX)"` || getRX=`calc "($cRX - $pRX)"`
    [ $cTX \< $pTX ] && getTX=`calc "$cTX + (0xFFFFFFFF - $pTX)"` || getTX=`calc "($cTX - $pTX)"`
    dlBytes=$(($getRX/$sec)); ulBytes=$(($getTX/$sec))
    [ $dlBytes -le "12000" -a $ulBytes -le "4000" ] && wanStatus="idle" || wanStatus="busy"

    getDLKbit=$(printf "%.0f\n" `calc $dlBytes*0.008`);        getULKbit=$(printf "%.0f\n" `calc $ulBytes*0.008`)
    getDLMbit=$(printf "%.2f\n" `calc $dlBytes*0.000008`);    getULMbit=$(printf "%.2f\n" `calc $ulBytes*0.000008`)
}

Create another shell script /jffs/bandwidth-env.sh with the following content:

export SPLUNK_AUTH="YOUR_SPLUNK_AUTH_KEY"
/jffs/bandwidth.sh

To test your script run it manually and confirm the data is showing in Splunk:

/jffs/bandwidth-env.sh

Schedule Your Script

To schedule your script, you can use the Scheduler (Administration/Schedule) in the router’s web admin UI. I have an automatic reboot scheduled at 4 AM, so I scheduled a custom script at 4:15 AM to run the bandwidth-env.sh script:

To start the script right away, spawn a process for it:

/jffs/bandwidth-env.sh &

Additional Info

Here’s a little bit of info on how the script works. The raw bandwidth data is read from /proc/net/dev.

Per redhad.com, /proc/net/dev "Lists the various network devices configured on the system, complete with transmit and receive statistics. This file displays the number of bytes each interface has sent and received, the number of packets inbound and outbound, the number of errors seen, the number of packets dropped, and more.”

For our purpose, we are interested in the first column which contains the cumulative number of bytes received by the interface, and the 10th column, which contains the number of bytes sent.

The script retrieves the current data, then sleeps for a number of seconds, and reads the updated data. The download/upload Mbit/s data is calculated by taking the difference and divide by the time elapsed. There’s also some logic to handle when the counters wrap around the max value back to zero.

Here’s how the data shows up in my Splunk Home dashboard:

The post Sending WAN/Internet Bandwidth Usage Data to Splunk from Tomato Routers using Splunk HEC appeared first on Chinh Do.

What had been still missing from the picture for me, is the ability to log, analyze, and visualize all the data that my smart home generated. I use Splunk (data capture and visualization tool) at work so I decided to give it a try at home and it’s worked out great.

Here’s a Splunk dashboard I created for my home, showing current and historical data from multiple data sources: energy meter, contact sensors, switches, weather data feed, Windows event logs, and some custom PowerShell scripts.

My SmartThings-based smart home setup:

• Samsung SmartThings Hub 2nd Gen
• Amazon Echo Devices
• Various ZigBee/Z-Wave devices
  • Samsung SmartThings GP-U999SJVLAAA Door & Window Multipurpose Sensors
  • Samsung SmartThings GP-U999SJVLBAA Motion Sensors
  • Samsung F-OUT-US-2 SmartThings Outlets
  • Other ZigBee/Z-Wave switches, dimmers, and plugs
  • Samsung ST-CEN-MOIS-1/FTR-US-2 Water Leak Sensors
  • Aeotec HEM G2 whole house energy monitor
  • First Alert ZCOMBO 2-in-1 Smoke Detector & Carbon Monoxide Alarm, Z-Wave
• PowerShell scripts to pull data from openweathermap.org & run/log periodic Internet speed tests.
• Splunk Free

Installing Splunk Free Edition

Download and install Splunk. You will start with the Enterprise version which comes with a 60-Day Trial. After that you can switch to the Free edition. Splunk Free allows indexing up to 500 MB of data per day which has been sufficient for my home logging needs. For my setup I installed Splunk on a 14-year old Windows box with a Intel Core2 Quad CPU Q6600 @2.40GHz – Splunk indexing/query performance has been pretty acceptable.

If your install was successful, you should be able to log into Splunk web by navigating to http://localhost:8000 (or replace localhost with your Splunk server hostname).

If you want to monitor other computers, install Splunk Universal Forwarder on each of those computers. I’ll go through how to configure the Universal Forwarders in a future post.

Enable the Splunk HTTP Event Collector (HEC)

The Splunk HTTP Event Collector is how we are going to send SmartThings events to Splunk. Follow the directions here to set it up. For my setup, I unchecked “Enable SSL”. Make sure you create an Authentication Token and note it down – you will need it later.

Send a test event so you can confirm that it’s working.

curl -k  https://localhost:8088/services/collector/event -H "Authorization: Splunk B5A79AAD-D822-46CC-80D1-819F80D7BFB0" -d '{"event": "hello world"}'

Install/Publish the Splunk HTTP Event Logger SmartThings SmartApp

A very cool benefit of using Samsung SmartThings is the ability to create your own SmartApps, or re-use SmartApp code written by others in the community. To send SmartThings events to Splunk, we will use SmartThingsSplunkLogger SmartApp code by Jason Hamilton/Brian Keifer:

• Go to https://graph.api.smartthings.com and log in
• Choose Locations and select your home location
• Click SmartApps, then “+ New SmartApp”
• Switch to the “From Code” tab
• Copy the source code from here and paste into the “From Code” box in your new SmartApp.

• Choose Create to create your SmartApp.
• Choose Publish/For Me to publish your SmartApp.
• See also:
  • community.smartthings.com discussion thread “Best way to collect logs from the hub?”

Enable the Splunk Logger SmartApp

• In your SmartThings mobile app (instructions here are for Android but should be similar for iOS), switch to the Automation tab.
• Choose “+Add a SmartApp”, scroll to the bottom and select My Apps, then Splunk HTTP Event Logger.
  • 10/28/2022: The “Add a SmartApp link has been moved in the latest app. It should now be in the Automations tab > + in the upper right > add routine > Discover tab and scroll to the bottom.

• Tap on each “Log these…” section and select all the devices you want to monitor.
• In the Splunk Server section:
  • Select Local Server
  • Fill in your Splunk Server hostname/IP
  • Leave Use SSL unchecked
  • Leave Splunk Port as the default 8088 (or change if you want)
  • Fill in the Splunk Authentication Token from earlier.
  • For “Assign a name”, type “Splunk Logger”
  • For “Set for specific mode(s), leave everything unchecked (default)

• Click Save at the top right corner.

At this point, if everything is working correctly, you should start seeing some SmartThings events in Splunk. You may need to wait 15 minutes for events to start coming in.

Build Your Splunk Dashboard

Splunk is fairly easy to use/learn so you should be able to learn the basics pretty quickly. If you are new to Splunk, try going through the Splunk Search Tutorial.

To get you started here are a few of the Splunk queries I use for my dashboard panels.

For the Current Power Consumption Radial Gauge, I use “stats latest(value)” to display the latest value from the Aeotec HEM G2 whole house energy monitor. I also limit the Time Range to the last 60 minutes since I am only interested in very recent data.

sourcetype=httpevent name=power deviceId="118c2de8-3256-44a6-96bd-d1547715fc92"
| stats latest(value)

For the Power Draw History panel, I use the timechart command to show a historical chart:

sourcetype=httpevent name=power deviceId="118c2de8-3256-44a6-96bd-d1547715fc92"
| timechart p50(value)

For the “Indoor Temperature” panel, the query uses a “by” cause to chart multiple series, one from each device. All of these devices are either contact sensors or moisture sensors that also have a temperature sensor built in.

sourcetype=httpevent name=temperature 
|  timechart avg(value) by device

Weather panels such as “Outdoor Temperature”, “Outdoor Humidity”, “Wind Speed” and Rain use data from openweathermap.org’s weather REST APIs. I wrote a PowerShell script that runs hourly to pull the data and appends to a log file, which is forwarded to Splunk. I will cover this in a future blog post.

For the “Contact Sensors & Switches” panel:

sourcetype=httpevent (name=contact OR name=switch)
| eval valueInt = case(value = "closed","1", value="open","2", value = "on","2", value="off","1")
|  stats latest(valueInt) by device

“CPU Load” and “Network Bytes per Sec” panels use data from Windows Event Logs.

CPU Load:
sourcetype="Perfmon:CPU Load" | timechart avg(Value) by host
Network Bytes:
sourcetype="Perfmon:Network Interface" (counter="Bytes Sent/sec" OR counter="Bytes Received/sec") | timechart sum(Value) by host

“WAN Speed” panel shows hourly Internet speed test results from a custom PowerShell script.

Hope this helps you to get started and move your smart home to the next level. In future posts, I will write more about installing and configuring Splunk Universal Forwarders, sending other types of data to Splunk, including Windows Events Log, weather data from openweathermap.org, and custom PowerShell scripts to log other data as well as querying Splunk data and send out alerts if certain conditions are met.

Troubleshooting Tips

If SmartThings events are not showing up in Splunk try looking at SmartThings Live Logging logs. Log into SmartThings Groovy IDE then choose Live Logging menu. When a SmartThing event occurs (such as when a door contact sensor is tripped), you should see a log entry written:

See Also

• Sending WAN/Internet Bandwidth Usage Data to Splunk from Tomato Routers using Splunk HEC

The post Monitor & Visualize Your SmartThings Smart Home with Splunk appeared first on Chinh Do.