skip to Main Content

Sending WAN/Internet Bandwidth Usage Data to Splunk from Tomato Routers using Splunk HEC

A while back I wrote about sending data from SmartThings and other home devices data to Splunk so I can monitor what goes on in my home via Splunk Dashboards. In addition to SmartThings devices, I also pulled data from other data sources such as network routers, Windows event logs, weather data retrieval scripts, etc.

To monitor our Internet bandwidth usage I wrote a Node.js program to scrape the data from the admin web UI for my Verizon Actiontec MI424WR router. Here‘s the code for that.

Last week I upgraded my internet to Verizon Fios Gigabit and with that upgrade, the Actiontec router was replaced with another router: a Netgear R7000 running Advanced Tomato (open source Linux-based firmware for Broadcom based Wi-fi routers). Advanced Tomato has a pretty click interface to monitor bandwidth, but I still want the data in my Splunk instance.

Luckily, Advanced Tomato runs a variant of Linux, so all I needed was a shell script to calculate bandwidth usage data and send to Splunk via the Splunk Http Event Collector.

I found a script by WaLLy3K that already had the bandwidth calculation logic and all I had to add was a little more code to send the data to Splunk.

Step-by-step Instructions

Enable JFFS Partition on Your Router

Enable JFFS Partition on your router so that you have permanent storage for your script. Otherwise if you saved your script in /tmp, it’ll be gone after the next reboot. Log into your router’s admin UI, choose Administration/JFFS, select Enabled and Save.

Create Your Script

SSH into your router and create a shell script at /jffs/bandwidth.sh with the content from here. Update the splunkUrl variable with your Splunk HEC URL. If you are not able to SSH, make sure you have SSH Daemon enabled under Administration/Admin Access.

For more info on installing Splunk HTTP Event Collection, see my previous post.

# this is just an excerpt of the code. For full code see 
# https://github.com/chinhdo/shell-scripts/blob/master/sh/bandwidth.sh

...
wan_iface=`nvram get wan_iface`
calc(){ awk "BEGIN { print $*}"; }    # Calculate floating point arithmetic using AWK instead of BC

checkWAN () {
    [ -z $1 ] && sec="1" || sec="$1"

    netdev=`grep "$wan_iface" /proc/net/dev`
    pRX=$(echo $netdev | cut -d' ' -f2)
    pTX=$(echo $netdev | cut -d' ' -f10)
    sleep $sec
    netdev=`grep "$wan_iface" /proc/net/dev`
    cRX=$(echo $netdev | cut -d' ' -f2)
    cTX=$(echo $netdev | cut -d' ' -f10)

    [ $cRX \< $pRX ] && getRX=`calc "$cRX + (0xFFFFFFFF - $pRX)"` || getRX=`calc "($cRX - $pRX)"`
    [ $cTX \< $pTX ] && getTX=`calc "$cTX + (0xFFFFFFFF - $pTX)"` || getTX=`calc "($cTX - $pTX)"`
    dlBytes=$(($getRX/$sec)); ulBytes=$(($getTX/$sec))
    [ $dlBytes -le "12000" -a $ulBytes -le "4000" ] && wanStatus="idle" || wanStatus="busy"

    getDLKbit=$(printf "%.0f\n" `calc $dlBytes*0.008`);        getULKbit=$(printf "%.0f\n" `calc $ulBytes*0.008`)
    getDLMbit=$(printf "%.2f\n" `calc $dlBytes*0.000008`);    getULMbit=$(printf "%.2f\n" `calc $ulBytes*0.000008`)
}

Create another shell script /jffs/bandwidth-env.sh with the following content:

export SPLUNK_AUTH="YOUR_SPLUNK_AUTH_KEY"
/jffs/bandwidth.sh

To test your script run it manually and confirm the data is showing in Splunk:

/jffs/bandwidth-env.sh
Splunk raw data

Schedule Your Script

To schedule your script, you can use the Scheduler (Administration/Schedule) in the router’s web admin UI. I have an automatic reboot scheduled at 4 AM, so I scheduled a custom script at 4:15 AM to run the bandwidth-env.sh script:

To start the script right away, spawn a process for it:

/jffs/bandwidth-env.sh &

Additional Info

Here’s a little bit of info on how the script works. The raw bandwidth data is read from /proc/net/dev.

Per redhad.com, /proc/net/dev "Lists the various network devices configured on the system, complete with transmit and receive statistics. This file displays the number of bytes each interface has sent and received, the number of packets inbound and outbound, the number of errors seen, the number of packets dropped, and more.”

Total bytes received and sent

For our purpose, we are interested in the first column which contains the cumulative number of bytes received by the interface, and the 10th column, which contains the number of bytes sent.

The script retrieves the current data, then sleeps for a number of seconds, and reads the updated data. The download/upload Mbit/s data is calculated by taking the difference and divide by the time elapsed. There’s also some logic to handle when the counters wrap around the max value back to zero.

Here’s how the data shows up in my Splunk Home dashboard:

Splunk Dashboard showing  WAN/Internet download/upload speed in Mbps

Best Demo 4K UHD Videos on YouTube

Just got a 4K Ultra HD TV/monitor and looking for things to watch/test your new monitor/TV? Try these YouTube videos.

From the HDR channel: LG’s demo video

The Ultimate High-resolution 4K movie in Super 35mm vol.01 | Alpha 7R II | Sony | α

Real 4K HDR: Cosmos Laundromat in HDR

Landscapes: Volume 4K (UHD)

TIMELAPSE OF THE FUTURE: A Journey to the End of Time (4K)

10 Incredible 4K (Ultra HD) Videos

Samsung 4K UHD demo video: Cityscape (with great song)

Demo 4K – Ultra HD

Japan in 8K 60fps

Transient – 4K, UHD, 1000FPS

Monitor & Visualize Your SmartThings Smart Home with Splunk

The smart home has gone through quite a convergence in the last few years. Modern protocols like Z-Wave & ZigBee, along with mart hubs, and smart assistants like Amazon Alexa, Google Home & Apple Siri are finally bringing everything together to make the smart home a practical and reliable reality.

What had been still missing from the picture for me, is the ability to log, analyze, and visualize all the data that my smart home generated. I use Splunk (data capture and visualization tool) at work so I decided to give it a try at home and it’s worked out great.

Here’s a Splunk dashboard I created for my home, showing current and historical data from multiple data sources: energy meter, contact sensors, switches, weather data feed, Windows event logs, and some custom PowerShell scripts.

My SmartThings-based smart home setup:

  • Samsung SmartThings Hub 2nd Gen
  • Amazon Echo Devices
  • Various ZigBee/Z-Wave devices
    • Samsung SmartThings GP-U999SJVLAAA Door & Window Multipurpose Sensors
    • Samsung SmartThings GP-U999SJVLBAA Motion Sensors
    • Samsung F-OUT-US-2 SmartThings Outlets
    • Other ZigBee/Z-Wave switches, dimmers, and plugs
    • Samsung ST-CEN-MOIS-1/FTR-US-2 Water Leak Sensors
    • Aeotec HEM G2 whole house energy monitor
    • First Alert ZCOMBO 2-in-1 Smoke Detector & Carbon Monoxide Alarm, Z-Wave
  • PowerShell scripts to pull data from openweathermap.org & run/log periodic Internet speed tests.
  • Splunk Free

Installing Splunk Free Edition

Download and install Splunk. You will start with the Enterprise version which comes with a 60-Day Trial. After that you can switch to the Free edition. Splunk Free allows indexing up to 500 MB of data per day which has been sufficient for my home logging needs. For my setup I installed Splunk on a 14-year old Windows box with a Intel Core2 Quad CPU Q6600 @2.40GHz – Splunk indexing/query performance has been pretty acceptable.

If your install was successful, you should be able to log into Splunk web by navigating to http://localhost:8000 (or replace localhost with your Splunk server hostname).

If you want to monitor other computers, install Splunk Universal Forwarder on each of those computers. I’ll go through how to configure the Universal Forwarders in a future post.

Read more

How to get free 2 GB of data on each line for Life on Verizon Wireless for about $40

Verizon Wireless is running a promo until 1/6/2016, that will give you 2 GB of free data for life if you are on an XL plan or larger, and you upgrade an existing device or add a new device.

Here’s how to you can take advantage of this promo and add the 2 GB of bonus data per line for life by spending about $50, even if you don’t have an upgrade available. To me, $40 for 2 GB of data per month for life is more than worth it.

Here’s how:

  • “Upgrade” your device to the cheapest smartphone available. Right now for me it would be the Droid Mini. The full price is $108.
  • If you are ordering online, make sure the confirmation page says you are getting 2 GB data bonus. If via a Verizon rep, confirm with her/him.
  • Activate the new Droid Mini.
  • Wait a day or two, and reactive your original phone, and sell the Droid Mini on eBay or the marketplace of your choice. You should get about $90 for it.
  • Total cost to you, excluding shipping, etc is about $40.

Disclaimer: while I believe this to work based on my own experience and available information. I cannot warranty that this will work for you.

Taking Control of Your Thermostat

Once in a while I come across a new product that solves a problem so elegantly that I just have to ask myself, why didn’t think of this before? It’s been very cold recently in the East Coast and when it gets very cold, my house’s gas heating system goes completely nuts. If I set the thermostat desired temperature to 70 degrees, the temperature in the bedroom will be in the roasting 80’s. The temperature differential depends how how cold it is outside, so I can’t just simply set the thermostat to a specific offset and forget either. I constantly need to get up in the middle of the night to adjust the thermostat downstairs when it gets cold outside. Why do I have to do this? I guess nobody told my house that we are in the 21st century.

image

So, the first thing I thought of is a remote control for the thermostat. Well, no surprise, they do make them. Apparently, my problem is fairly common for two-story homes with a single HVAC system. This Lux TX9000RF Programmable Thermostat with Remote looked very promising to me. A product like this would allow adjusting the thermostat temperature from anywhere in the house.

That still requires some work however. Hmm… what if there is a thermostat that can read the current temperature from a remote sensor? Bingo: they make those too. There are not many to be found, and after searching around, I decided to go for the  Honeywell YTHX9321R5003 Prestige HD Thermostat Kit and I’ve been very happy with the result so far. This kit is expensive, but very well made and it works as advertised. It also looks very nice. The kit includes the thermostat, a remote control/sensor, and an outdoor sensor. This kit is in Honeywell Pro Install line, which means it’s sold mainly through HVAC contractors and installers. I found the installation process only slightly more complicated than a regular programmable thermostat. The only thing you need to watch out for is that this thermostat requires a 24vac Common wire (commonly black in color), which may not be available in your setup. If that is the case, then you will need to run/fish a new wire from your furnace – a pretty big job.

Now with this cool new gadget hooked up and everything humming, all I have to do is bring the remote with me to the bedroom and push the button on it named “Read temp from this device” and I am set for the night. If I ever want to tweak the temperature for some reason, I can do it right there with the remote. If only everything else was this easy!

clip_image001

Windows Mobile 6.1 for Samsung SCH-i760

Good news for i760 owners: Windows Mobile 6.1 update is now available.

This update fixes one major annoyance: support for SDHC cards with more than 2GB. I use my i760 as a music player and it’s kind of tough to have to fit my music selection into 2GB (or 4GB if you use the hack but didn’t want to use a hack).

Some features of note in WM 6.1:

  • Support for SDHC cards  beyond 2GB. 8GB cards seem to be working fine for people.
  • Threaded SMS reader.
  • View YouTube videos (m.youtube.com).

Download the update from Verizon here.

More information here.

Verizon SCH-i760 Upgrade Tool

A word of warning:  the upgrade utility provided by Verizon/Samsung looks like a major piece of rushware. Many people reported no problems with the upgrade, but others reported of bricked phones, partially upgraded phones, and having to try the upgrade process multiple times to get it to work.

Finds of the Week – May 31, 2008

Programming

.NET/C#

Software and Tools

  • Google Maps Street View is in Richmond.

    Google Street View Richmond Virginia Goo

    It looks like the Street View images were taken around September 2007 because according to IMDB, the movie Game Plan was playing in the US starting 9/23/2007:

    Google Street View Richmond

  • Google Maps adds user-created photos, videos, maps. Via CNET.

    Google Maps Goog

Something Different

Finds of the Week – January 20, 2008

.NET, C#, Programming

Software and Tools

  • Jeff Atwood shared the Top Five Browser Shortcuts Everyone Should Know. I didn’t know about the middle mouse button browser shortcut. Very useful.
  • Reading Nazmul Idris’ post on Microsoft OneNote 2007 prompted me to try it myself…. and I loved it. It’s a great application to keep track of notes, journal entries, work logs, etc. It’s going to be an essential app for me from now on.

    OneNote 

    I especially like offline mode support for USB flash drives. You can keep your Notebooks on a flash memory card. OneNote automatically synchronizes its local cache with the flash card when the card is inserted. When not inserted, you can still work on the local cached copy.

    What I don’t like: no VBA macro support. One of the first thing I tried was pressing ALT+F11 to bring up the VBA IDE… nothing happened. A quick search in Online Help confirmed my sinking feeling: no VBA support. Supposedly, you can write add-ins.

Windows Mobile/Pocket PC

Gadgets

Something different

  • Watch dolphins blowing circles made of air bubbles.

Finds of the Week – Nov 16, 2007

.NET

Software and Tools

Tech and Gadgets

  • Dish Network enables ethernet ports on ViP622 HD DVRs. The main feature that the broadband connection brings is video on demand. The ethernet connection also allows the box to call home via the Internet instead of using the phone line.

Miscellaneous

  • View popular “sightings” on Google Maps, Windows Live Maps, and Yahoo Maps at Virtual Globetrotting

And Now, Something Different

I cut a kiwi fruit in halves the other day and this was what I saw:

Kiki Fruit Heart

Back To Top