skip to Main Content

Sending WAN/Internet Bandwidth Usage Data to Splunk from Tomato Routers using Splunk HEC

A while back I wrote about sending data from SmartThings and other home devices data to Splunk so I can monitor what goes on in my home via Splunk Dashboards. In addition to SmartThings devices, I also pulled data from other data sources such as network routers, Windows event logs, weather data retrieval scripts, etc.

To monitor our Internet bandwidth usage I wrote a Node.js program to scrape the data from the admin web UI for my Verizon Actiontec MI424WR router. Here‘s the code for that.

Last week I upgraded my internet to Verizon Fios Gigabit and with that upgrade, the Actiontec router was replaced with another router: a Netgear R7000 running Advanced Tomato (open source Linux-based firmware for Broadcom based Wi-fi routers). Advanced Tomato has a pretty click interface to monitor bandwidth, but I still want the data in my Splunk instance.

Luckily, Advanced Tomato runs a variant of Linux, so all I needed was a shell script to calculate bandwidth usage data and send to Splunk via the Splunk Http Event Collector.

I found a script by WaLLy3K that already had the bandwidth calculation logic and all I had to add was a little more code to send the data to Splunk.

Step-by-step Instructions

Enable JFFS Partition on Your Router

Enable JFFS Partition on your router so that you have permanent storage for your script. Otherwise if you saved your script in /tmp, it’ll be gone after the next reboot. Log into your router’s admin UI, choose Administration/JFFS, select Enabled and Save.

Create Your Script

SSH into your router and create a shell script at /jffs/bandwidth.sh with the content from here. Update the splunkUrl variable with your Splunk HEC URL. If you are not able to SSH, make sure you have SSH Daemon enabled under Administration/Admin Access.

For more info on installing Splunk HTTP Event Collection, see my previous post.

# this is just an excerpt of the code. For full code see 
# https://github.com/chinhdo/shell-scripts/blob/master/sh/bandwidth.sh

...
wan_iface=`nvram get wan_iface`
calc(){ awk "BEGIN { print $*}"; }    # Calculate floating point arithmetic using AWK instead of BC

checkWAN () {
    [ -z $1 ] && sec="1" || sec="$1"

    netdev=`grep "$wan_iface" /proc/net/dev`
    pRX=$(echo $netdev | cut -d' ' -f2)
    pTX=$(echo $netdev | cut -d' ' -f10)
    sleep $sec
    netdev=`grep "$wan_iface" /proc/net/dev`
    cRX=$(echo $netdev | cut -d' ' -f2)
    cTX=$(echo $netdev | cut -d' ' -f10)

    [ $cRX \< $pRX ] && getRX=`calc "$cRX + (0xFFFFFFFF - $pRX)"` || getRX=`calc "($cRX - $pRX)"`
    [ $cTX \< $pTX ] && getTX=`calc "$cTX + (0xFFFFFFFF - $pTX)"` || getTX=`calc "($cTX - $pTX)"`
    dlBytes=$(($getRX/$sec)); ulBytes=$(($getTX/$sec))
    [ $dlBytes -le "12000" -a $ulBytes -le "4000" ] && wanStatus="idle" || wanStatus="busy"

    getDLKbit=$(printf "%.0f\n" `calc $dlBytes*0.008`);        getULKbit=$(printf "%.0f\n" `calc $ulBytes*0.008`)
    getDLMbit=$(printf "%.2f\n" `calc $dlBytes*0.000008`);    getULMbit=$(printf "%.2f\n" `calc $ulBytes*0.000008`)
}

Create another shell script /jffs/bandwidth-env.sh with the following content:

export SPLUNK_AUTH="YOUR_SPLUNK_AUTH_KEY"
/jffs/bandwidth.sh

To test your script run it manually and confirm the data is showing in Splunk:

/jffs/bandwidth-env.sh
Splunk raw data

Schedule Your Script

To schedule your script, you can use the Scheduler (Administration/Schedule) in the router’s web admin UI. I have an automatic reboot scheduled at 4 AM, so I scheduled a custom script at 4:15 AM to run the bandwidth-env.sh script:

To start the script right away, spawn a process for it:

/jffs/bandwidth-env.sh &

Additional Info

Here’s a little bit of info on how the script works. The raw bandwidth data is read from /proc/net/dev.

Per redhad.com, /proc/net/dev "Lists the various network devices configured on the system, complete with transmit and receive statistics. This file displays the number of bytes each interface has sent and received, the number of packets inbound and outbound, the number of errors seen, the number of packets dropped, and more.”

Total bytes received and sent

For our purpose, we are interested in the first column which contains the cumulative number of bytes received by the interface, and the 10th column, which contains the number of bytes sent.

The script retrieves the current data, then sleeps for a number of seconds, and reads the updated data. The download/upload Mbit/s data is calculated by taking the difference and divide by the time elapsed. There’s also some logic to handle when the counters wrap around the max value back to zero.

Here’s how the data shows up in my Splunk Home dashboard:

Splunk Dashboard showing  WAN/Internet download/upload speed in Mbps

Best Demo 4K UHD Videos on YouTube

Just got a 4K Ultra HD TV/monitor and looking for things to watch/test your new monitor/TV? Try these YouTube videos.

From the HDR channel: LG’s demo video

The Ultimate High-resolution 4K movie in Super 35mm vol.01 | Alpha 7R II | Sony | α

Real 4K HDR: Cosmos Laundromat in HDR

Landscapes: Volume 4K (UHD)

TIMELAPSE OF THE FUTURE: A Journey to the End of Time (4K)

10 Incredible 4K (Ultra HD) Videos

Samsung 4K UHD demo video: Cityscape (with great song)

Demo 4K – Ultra HD

Japan in 8K 60fps

Transient – 4K, UHD, 1000FPS

Monitor & Visualize Your SmartThings Smart Home with Splunk

The smart home has gone through quite a convergence in the last few years. Modern protocols like Z-Wave & ZigBee, along with mart hubs, and smart assistants like Amazon Alexa, Google Home & Apple Siri are finally bringing everything together to make the smart home a practical and reliable reality.

What had been still missing from the picture for me, is the ability to log, analyze, and visualize all the data that my smart home generated. I use Splunk (data capture and visualization tool) at work so I decided to give it a try at home and it’s worked out great.

Here’s a Splunk dashboard I created for my home, showing current and historical data from multiple data sources: energy meter, contact sensors, switches, weather data feed, Windows event logs, and some custom PowerShell scripts.

My SmartThings-based smart home setup:

  • Samsung SmartThings Hub 2nd Gen
  • Amazon Echo Devices
  • Various ZigBee/Z-Wave devices
    • Samsung SmartThings GP-U999SJVLAAA Door & Window Multipurpose Sensors
    • Samsung SmartThings GP-U999SJVLBAA Motion Sensors
    • Samsung F-OUT-US-2 SmartThings Outlets
    • Other ZigBee/Z-Wave switches, dimmers, and plugs
    • Samsung ST-CEN-MOIS-1/FTR-US-2 Water Leak Sensors
    • Aeotec HEM G2 whole house energy monitor
    • First Alert ZCOMBO 2-in-1 Smoke Detector & Carbon Monoxide Alarm, Z-Wave
  • PowerShell scripts to pull data from openweathermap.org & run/log periodic Internet speed tests.
  • Splunk Free

Installing Splunk Free Edition

Download and install Splunk. You will start with the Enterprise version which comes with a 60-Day Trial. After that you can switch to the Free edition. Splunk Free allows indexing up to 500 MB of data per day which has been sufficient for my home logging needs. For my setup I installed Splunk on a 14-year old Windows box with a Intel Core2 Quad CPU Q6600 @2.40GHz – Splunk indexing/query performance has been pretty acceptable.

If your install was successful, you should be able to log into Splunk web by navigating to http://localhost:8000 (or replace localhost with your Splunk server hostname).

If you want to monitor other computers, install Splunk Universal Forwarder on each of those computers. I’ll go through how to configure the Universal Forwarders in a future post.

Read more

How to get free 2 GB of data on each line for Life on Verizon Wireless for about $40

Verizon Wireless is running a promo until 1/6/2016, that will give you 2 GB of free data for life if you are on an XL plan or larger, and you upgrade an existing device or add a new device.

Here’s how to you can take advantage of this promo and add the 2 GB of bonus data per line for life by spending about $50, even if you don’t have an upgrade available. To me, $40 for 2 GB of data per month for life is more than worth it.

Here’s how:

  • “Upgrade” your device to the cheapest smartphone available. Right now for me it would be the Droid Mini. The full price is $108.
  • If you are ordering online, make sure the confirmation page says you are getting 2 GB data bonus. If via a Verizon rep, confirm with her/him.
  • Activate the new Droid Mini.
  • Wait a day or two, and reactive your original phone, and sell the Droid Mini on eBay or the marketplace of your choice. You should get about $90 for it.
  • Total cost to you, excluding shipping, etc is about $40.

Disclaimer: while I believe this to work based on my own experience and available information. I cannot warranty that this will work for you.

Make all of your home servers reachable by Remote Desktop

If you have more than one servers/PCs running at home behind the internet router, you can normally set up the router port forwarding to forward Remote Desktop traffic to just one server. However, by assigning different incoming ports for different servers, you can connect to each server individually.

Basically, use a different port for each server. Then configure the port forwarding on the router to router each port to the appropriate server on 3389. This assumes that your router supports different ranges for external and internal ports.

For example, for my home setup, I use three different port numbers for the 3 servers I have at home:

RemoteDesktopPortForwarding

By using non-standard ports, there’s also a side benefit of a little extra security.

In Remote Desktop Client, you would add the port number to the Computer name like this:

RemoteDesktopClient

 

Happy Remote Desktoping!

Managing, Tagging, Organizing Videos with Picasa

I enjoy making taking and making short videos. My kids are usually featured in the videos in one way or another. Last year we went on a month-long vacation to Vietnam and I recorded a huge number of raw videos. With so many raw takes to work with, using Windows Explorer and browsing around without some system of tagging, keyword searching, etc. becomes extremely tedious and unmanageable.

So I started to look for a way to manage/categorize all of these raw videos and I think I’ve found a pretty good and free solution: Picasa (from Google).

The current version of Picasa for Windows (version 3.8) allows you to specify titles for images and videos, as well as assigning various tags to them. The user interface is very user-friendly. And the actual searches are typical Google-lighting-fast.

Using Picasa as a video manager

What I do is I go through each raw video take, give it a title and assign some tags. Then later I am able to very quickly search for videos. For example, if I need a take involving a dolly pan to the right, I just type “dolly pan” into Picasa and instantly the takes with those keywords come up.

Instant Messaging Etiquette for the Workplace

The use of instant messaging at the workplace is very prevalent these days. While instant messaging is very convenient for everyday communication, don’t treat it the same as email, face-to-face meetings, or phone calls. There are several significant differences between instant messaging and traditional forms of communications:

  • Instant messaging is not necessarily one-one-one. At any given moment, one person may be engaged in several simultaneous instant messaging conversations.
  • Even though your messages will be displayed on the recipient’s screen immediately, the recipient may not be able to read the messages and respond immediately.
  • With most instant messaging applications, the recipient cannot read your messages until you press the Enter key to send it. This limitation inherently makes instant messaging significantly slower than voice conversations.
  • Most people can speak faster than they can type.

Here are some guidelines on basic/everyday instant messaging etiquette that will help you and your co-workers make the most out of this communication medium.

Instant Messaging apps - AIM, MSN Messenger, Windows Live Messenger, Yahoo Messenger, Google Talk, ICQ 

Include Relevant Info/Questions in The First Message

Avoid typing greetings or non-essential messages separately first. Include the question or relevant information in your first message.

Not-so-good examples (avoid this)

Conversation 1

  • Jane (10:54:50 AM): Hi
  • Mike (10:55:01 AM): Hello
  • Jane (10:55:06 AM): The test server will be restarted in 1 minute.
  • Mike (10:56:11 AM): Thanks for the info

Conversation 2

  • Jane (2:15:08 PM): Good afternoon
  • Mike (2:15:13 PM): Good afternoon
  • Jane (2:15:13 PM): Do you have a few minutes to talk on the phone re project A?
  • Mike (2:15:18 PM): Sure, let me call you.

Conversation 3

  • Jane (4:03:30 PM): You there?
  • Mike (4:03:35 PM): Yes
  • Jane (4:03:42 PM): Can we have a short team meeting in conf room A
  • Mike (4:03:50 PM): Sure. Be there in 5.

Good examples

Conversation 1

  • Jane (10:54:50 AM): Hi, the test server will be restarted in 1 minute.
  • Mike (10:54:55 AM): Thanks for the info

Conversation 2

  • Jane (2:15:08 PM): Good afternoon, do you have a few minutes to talk on the phone re project A?
  • Mike (2:15:13 PM): Sure I will call you.

Conversation 3

  • Jane (4:03:30 PM): Can we have a short team meeting in conf room A?
  • Mike (4:03:35 PM): Sure, be right there.

Every time you send a message, the recipient is disrupted from whatever he/she is doing. The more you can delay this disruption the better, even if it’s only seconds. It’s also not necessary to ask the recipient if he/she is there. That’s what the online status (away/available) is for. You can just type your message or ask your question. One exception to this would be if your message contains sensitive information.

Send Complete Messages

Good:

  • Jane (9:15:23 AM): Hi, all integration tests are failing in the integration environment for the admin user group. Can you take a look?”

No so good:

  • Jane (9:15:23 AM): Hi, all integration tests are failing…
  • Jane (9:15:27 AM): the integration environment…
  • Jane (9:15:32 AM): for the admin user group…”
  • Jane (9:15:38 AM): Can you take a look?

If you have to send several sentences in sequence, compose them in a separate editor first (notepad would do), then type them out quickly in succession. That way the recipient does not have to wait for you while you type your next message, make editing corrections, etc.

Do Not Expect Responses Immediately

Unless the recipient reports to you, do not expect a response immediately after you type a message. The recipient may be in the middle of five other instant message conversations, on the phone, or working on something else more important. If the recipient’s status is not set to “Away”, and you have not received a response after a few minutes, it’s ok to ping him/her again. Maybe they forgot about your message.

Update Your Status and Respect Others’ Status

If you need to step out, change your status to “Away”, or “In a Meeting”, or “Back in an Hour”, or whatever is appropriate for the situation. This tells everyone else that you are not available to respond to messages immediately and save them from having to wait for your responses.

Pick Up the Phone

If the instant messaging conversation starts to go into lots of details and may make take longer than a few minutes, consider picking up the phone and continue the conversation there.

Show Your Papers!

They say a man’s home is his castle, and since my computer is my virtual home, I must have complete control over what goes on in it.  When I see a stranger person walking around in my yard, he’s better be ready to tell me who he is, what company he works for, why he’s in there. Likewise, when I see a strange window running in my computer, I must have the ability to easily tell what it is, who makes it, when it was installed, etc.

Yes, one would think that being able to quickly identify any running window would be a basic feature of any modern so called window operation system. It’s 25 years after the first release of Microsoft Windows, and the sad truth is that you still often cannot easily identify running windows.

Look at the example below. If you are not familiar with this utility, and you came back to your laptop seeing this, would you know what it’s about? Should you click Yes or No? Is this a legitimate application, or something more sinister?

image 

The first obvious problem is the missing message. That’s forgivable however. Bugs happen, files get corrupted, language resource files go missing, etc. What’s not acceptable is for the Windows OS not to provide any method to identify misbehaving windows.

So how about it Microsoft? Let us easily find out identifying information about any running Windows. Perhaps with with a click of a button, we can see:

  • Name of owning application/process
  • Name of vendor (if available)
  • Folder where executable resides
  • Date the application was installed
  • User who installed the application
  • If the user didn’t run the application himself, identify the parent process or service that launches the application (shortcut in Startup folder, registry, etc.)
  • Available code signatures

For now, if you want to identify any visible window, use Sysinternals’ Process Explorer. Drag the “Find Window’s Process” icon and drop it on top of the target window and Process Explorer will highlight the owner process in its window. From there, you can get the executable name, company name, folder location, etc.

Process Explorer Find Window's Process

Taking Control of Your Thermostat

Once in a while I come across a new product that solves a problem so elegantly that I just have to ask myself, why didn’t think of this before? It’s been very cold recently in the East Coast and when it gets very cold, my house’s gas heating system goes completely nuts. If I set the thermostat desired temperature to 70 degrees, the temperature in the bedroom will be in the roasting 80’s. The temperature differential depends how how cold it is outside, so I can’t just simply set the thermostat to a specific offset and forget either. I constantly need to get up in the middle of the night to adjust the thermostat downstairs when it gets cold outside. Why do I have to do this? I guess nobody told my house that we are in the 21st century.

image

So, the first thing I thought of is a remote control for the thermostat. Well, no surprise, they do make them. Apparently, my problem is fairly common for two-story homes with a single HVAC system. This Lux TX9000RF Programmable Thermostat with Remote looked very promising to me. A product like this would allow adjusting the thermostat temperature from anywhere in the house.

That still requires some work however. Hmm… what if there is a thermostat that can read the current temperature from a remote sensor? Bingo: they make those too. There are not many to be found, and after searching around, I decided to go for the  Honeywell YTHX9321R5003 Prestige HD Thermostat Kit and I’ve been very happy with the result so far. This kit is expensive, but very well made and it works as advertised. It also looks very nice. The kit includes the thermostat, a remote control/sensor, and an outdoor sensor. This kit is in Honeywell Pro Install line, which means it’s sold mainly through HVAC contractors and installers. I found the installation process only slightly more complicated than a regular programmable thermostat. The only thing you need to watch out for is that this thermostat requires a 24vac Common wire (commonly black in color), which may not be available in your setup. If that is the case, then you will need to run/fish a new wire from your furnace – a pretty big job.

Now with this cool new gadget hooked up and everything humming, all I have to do is bring the remote with me to the bedroom and push the button on it named “Read temp from this device” and I am set for the night. If I ever want to tweak the temperature for some reason, I can do it right there with the remote. If only everything else was this easy!

clip_image001

Back To Top