To monitor our Internet bandwidth usage I wrote a Node.js program to scrape the data from the admin web UI for my Verizon Actiontec MI424WR router. Here‘s the code for that.

Last week I upgraded my internet to Verizon Fios Gigabit and with that upgrade, the Actiontec router was replaced with another router: a Netgear R7000 running Advanced Tomato (open source Linux-based firmware for Broadcom based Wi-fi routers). Advanced Tomato has a pretty click interface to monitor bandwidth, but I still want the data in my Splunk instance.

Luckily, Advanced Tomato runs a variant of Linux, so all I needed was a shell script to calculate bandwidth usage data and send to Splunk via the Splunk Http Event Collector.

I found a script by WaLLy3K that already had the bandwidth calculation logic and all I had to add was a little more code to send the data to Splunk.

Step-by-step Instructions

Enable JFFS Partition on Your Router

Enable JFFS Partition on your router so that you have permanent storage for your script. Otherwise if you saved your script in /tmp, it’ll be gone after the next reboot. Log into your router’s admin UI, choose Administration/JFFS, select Enabled and Save.

Create Your Script

SSH into your router and create a shell script at /jffs/bandwidth.sh with the content from here. Update the splunkUrl variable with your Splunk HEC URL. If you are not able to SSH, make sure you have SSH Daemon enabled under Administration/Admin Access.

For more info on installing Splunk HTTP Event Collection, see my previous post.

# this is just an excerpt of the code. For full code see 
# https://github.com/chinhdo/shell-scripts/blob/master/sh/bandwidth.sh

...
wan_iface=`nvram get wan_iface`
calc(){ awk "BEGIN { print $*}"; }    # Calculate floating point arithmetic using AWK instead of BC

checkWAN () {
    [ -z $1 ] && sec="1" || sec="$1"

    netdev=`grep "$wan_iface" /proc/net/dev`
    pRX=$(echo $netdev | cut -d' ' -f2)
    pTX=$(echo $netdev | cut -d' ' -f10)
    sleep $sec
    netdev=`grep "$wan_iface" /proc/net/dev`
    cRX=$(echo $netdev | cut -d' ' -f2)
    cTX=$(echo $netdev | cut -d' ' -f10)

    [ $cRX \< $pRX ] && getRX=`calc "$cRX + (0xFFFFFFFF - $pRX)"` || getRX=`calc "($cRX - $pRX)"`
    [ $cTX \< $pTX ] && getTX=`calc "$cTX + (0xFFFFFFFF - $pTX)"` || getTX=`calc "($cTX - $pTX)"`
    dlBytes=$(($getRX/$sec)); ulBytes=$(($getTX/$sec))
    [ $dlBytes -le "12000" -a $ulBytes -le "4000" ] && wanStatus="idle" || wanStatus="busy"

    getDLKbit=$(printf "%.0f\n" `calc $dlBytes*0.008`);        getULKbit=$(printf "%.0f\n" `calc $ulBytes*0.008`)
    getDLMbit=$(printf "%.2f\n" `calc $dlBytes*0.000008`);    getULMbit=$(printf "%.2f\n" `calc $ulBytes*0.000008`)
}

Create another shell script /jffs/bandwidth-env.sh with the following content:

export SPLUNK_AUTH="YOUR_SPLUNK_AUTH_KEY"
/jffs/bandwidth.sh

To test your script run it manually and confirm the data is showing in Splunk:

/jffs/bandwidth-env.sh

Schedule Your Script

To schedule your script, you can use the Scheduler (Administration/Schedule) in the router’s web admin UI. I have an automatic reboot scheduled at 4 AM, so I scheduled a custom script at 4:15 AM to run the bandwidth-env.sh script:

To start the script right away, spawn a process for it:

/jffs/bandwidth-env.sh &

Additional Info

Here’s a little bit of info on how the script works. The raw bandwidth data is read from /proc/net/dev.

Per redhad.com, /proc/net/dev "Lists the various network devices configured on the system, complete with transmit and receive statistics. This file displays the number of bytes each interface has sent and received, the number of packets inbound and outbound, the number of errors seen, the number of packets dropped, and more.”

For our purpose, we are interested in the first column which contains the cumulative number of bytes received by the interface, and the 10th column, which contains the number of bytes sent.

The script retrieves the current data, then sleeps for a number of seconds, and reads the updated data. The download/upload Mbit/s data is calculated by taking the difference and divide by the time elapsed. There’s also some logic to handle when the counters wrap around the max value back to zero.

Here’s how the data shows up in my Splunk Home dashboard:

The post Sending WAN/Internet Bandwidth Usage Data to Splunk from Tomato Routers using Splunk HEC appeared first on Chinh Do.

From the HDR channel: LG’s demo video

The Ultimate High-resolution 4K movie in Super 35mm vol.01 | Alpha 7R II | Sony | α

Real 4K HDR: Cosmos Laundromat in HDR

Landscapes: Volume 4K (UHD)

TIMELAPSE OF THE FUTURE: A Journey to the End of Time (4K)

10 Incredible 4K (Ultra HD) Videos

Samsung 4K UHD demo video: Cityscape (with great song)

Demo 4K – Ultra HD

Japan in 8K 60fps

Transient – 4K, UHD, 1000FPS

The post Best Demo 4K UHD Videos on YouTube appeared first on Chinh Do.

What had been still missing from the picture for me, is the ability to log, analyze, and visualize all the data that my smart home generated. I use Splunk (data capture and visualization tool) at work so I decided to give it a try at home and it’s worked out great.

Here’s a Splunk dashboard I created for my home, showing current and historical data from multiple data sources: energy meter, contact sensors, switches, weather data feed, Windows event logs, and some custom PowerShell scripts.

My SmartThings-based smart home setup:

• Samsung SmartThings Hub 2nd Gen
• Amazon Echo Devices
• Various ZigBee/Z-Wave devices
  • Samsung SmartThings GP-U999SJVLAAA Door & Window Multipurpose Sensors
  • Samsung SmartThings GP-U999SJVLBAA Motion Sensors
  • Samsung F-OUT-US-2 SmartThings Outlets
  • Other ZigBee/Z-Wave switches, dimmers, and plugs
  • Samsung ST-CEN-MOIS-1/FTR-US-2 Water Leak Sensors
  • Aeotec HEM G2 whole house energy monitor
  • First Alert ZCOMBO 2-in-1 Smoke Detector & Carbon Monoxide Alarm, Z-Wave
• PowerShell scripts to pull data from openweathermap.org & run/log periodic Internet speed tests.
• Splunk Free

Installing Splunk Free Edition

Download and install Splunk. You will start with the Enterprise version which comes with a 60-Day Trial. After that you can switch to the Free edition. Splunk Free allows indexing up to 500 MB of data per day which has been sufficient for my home logging needs. For my setup I installed Splunk on a 14-year old Windows box with a Intel Core2 Quad CPU Q6600 @2.40GHz – Splunk indexing/query performance has been pretty acceptable.

If your install was successful, you should be able to log into Splunk web by navigating to http://localhost:8000 (or replace localhost with your Splunk server hostname).

If you want to monitor other computers, install Splunk Universal Forwarder on each of those computers. I’ll go through how to configure the Universal Forwarders in a future post.

Enable the Splunk HTTP Event Collector (HEC)

The Splunk HTTP Event Collector is how we are going to send SmartThings events to Splunk. Follow the directions here to set it up. For my setup, I unchecked “Enable SSL”. Make sure you create an Authentication Token and note it down – you will need it later.

Send a test event so you can confirm that it’s working.

curl -k  https://localhost:8088/services/collector/event -H "Authorization: Splunk B5A79AAD-D822-46CC-80D1-819F80D7BFB0" -d '{"event": "hello world"}'

Install/Publish the Splunk HTTP Event Logger SmartThings SmartApp

A very cool benefit of using Samsung SmartThings is the ability to create your own SmartApps, or re-use SmartApp code written by others in the community. To send SmartThings events to Splunk, we will use SmartThingsSplunkLogger SmartApp code by Jason Hamilton/Brian Keifer:

• Go to https://graph.api.smartthings.com and log in
• Choose Locations and select your home location
• Click SmartApps, then “+ New SmartApp”
• Switch to the “From Code” tab
• Copy the source code from here and paste into the “From Code” box in your new SmartApp.

• Choose Create to create your SmartApp.
• Choose Publish/For Me to publish your SmartApp.
• See also:
  • community.smartthings.com discussion thread “Best way to collect logs from the hub?”

Enable the Splunk Logger SmartApp

• In your SmartThings mobile app (instructions here are for Android but should be similar for iOS), switch to the Automation tab.
• Choose “+Add a SmartApp”, scroll to the bottom and select My Apps, then Splunk HTTP Event Logger.
  • 10/28/2022: The “Add a SmartApp link has been moved in the latest app. It should now be in the Automations tab > + in the upper right > add routine > Discover tab and scroll to the bottom.

• Tap on each “Log these…” section and select all the devices you want to monitor.
• In the Splunk Server section:
  • Select Local Server
  • Fill in your Splunk Server hostname/IP
  • Leave Use SSL unchecked
  • Leave Splunk Port as the default 8088 (or change if you want)
  • Fill in the Splunk Authentication Token from earlier.
  • For “Assign a name”, type “Splunk Logger”
  • For “Set for specific mode(s), leave everything unchecked (default)

• Click Save at the top right corner.

At this point, if everything is working correctly, you should start seeing some SmartThings events in Splunk. You may need to wait 15 minutes for events to start coming in.

Build Your Splunk Dashboard

Splunk is fairly easy to use/learn so you should be able to learn the basics pretty quickly. If you are new to Splunk, try going through the Splunk Search Tutorial.

To get you started here are a few of the Splunk queries I use for my dashboard panels.

For the Current Power Consumption Radial Gauge, I use “stats latest(value)” to display the latest value from the Aeotec HEM G2 whole house energy monitor. I also limit the Time Range to the last 60 minutes since I am only interested in very recent data.

sourcetype=httpevent name=power deviceId="118c2de8-3256-44a6-96bd-d1547715fc92"
| stats latest(value)

For the Power Draw History panel, I use the timechart command to show a historical chart:

sourcetype=httpevent name=power deviceId="118c2de8-3256-44a6-96bd-d1547715fc92"
| timechart p50(value)

For the “Indoor Temperature” panel, the query uses a “by” cause to chart multiple series, one from each device. All of these devices are either contact sensors or moisture sensors that also have a temperature sensor built in.

sourcetype=httpevent name=temperature 
|  timechart avg(value) by device

Weather panels such as “Outdoor Temperature”, “Outdoor Humidity”, “Wind Speed” and Rain use data from openweathermap.org’s weather REST APIs. I wrote a PowerShell script that runs hourly to pull the data and appends to a log file, which is forwarded to Splunk. I will cover this in a future blog post.

For the “Contact Sensors & Switches” panel:

sourcetype=httpevent (name=contact OR name=switch)
| eval valueInt = case(value = "closed","1", value="open","2", value = "on","2", value="off","1")
|  stats latest(valueInt) by device

“CPU Load” and “Network Bytes per Sec” panels use data from Windows Event Logs.

CPU Load:
sourcetype="Perfmon:CPU Load" | timechart avg(Value) by host
Network Bytes:
sourcetype="Perfmon:Network Interface" (counter="Bytes Sent/sec" OR counter="Bytes Received/sec") | timechart sum(Value) by host

“WAN Speed” panel shows hourly Internet speed test results from a custom PowerShell script.

Hope this helps you to get started and move your smart home to the next level. In future posts, I will write more about installing and configuring Splunk Universal Forwarders, sending other types of data to Splunk, including Windows Events Log, weather data from openweathermap.org, and custom PowerShell scripts to log other data as well as querying Splunk data and send out alerts if certain conditions are met.

Troubleshooting Tips

If SmartThings events are not showing up in Splunk try looking at SmartThings Live Logging logs. Log into SmartThings Groovy IDE then choose Live Logging menu. When a SmartThing event occurs (such as when a door contact sensor is tripped), you should see a log entry written:

See Also

• Sending WAN/Internet Bandwidth Usage Data to Splunk from Tomato Routers using Splunk HEC

The post Monitor & Visualize Your SmartThings Smart Home with Splunk appeared first on Chinh Do.

Here’s how to you can take advantage of this promo and add the 2 GB of bonus data per line for life by spending about $50, even if you don’t have an upgrade available. To me, $40 for 2 GB of data per month for life is more than worth it.

Here’s how:

• “Upgrade” your device to the cheapest smartphone available. Right now for me it would be the Droid Mini. The full price is $108.
• If you are ordering online, make sure the confirmation page says you are getting 2 GB data bonus. If via a Verizon rep, confirm with her/him.
• Activate the new Droid Mini.
• Wait a day or two, and reactive your original phone, and sell the Droid Mini on eBay or the marketplace of your choice. You should get about $90 for it.
• Total cost to you, excluding shipping, etc is about $40.

Disclaimer: while I believe this to work based on my own experience and available information. I cannot warranty that this will work for you.

The post How to get free 2 GB of data on each line for Life on Verizon Wireless for about $40 appeared first on Chinh Do.

Basically, use a different port for each server. Then configure the port forwarding on the router to router each port to the appropriate server on 3389. This assumes that your router supports different ranges for external and internal ports.

For example, for my home setup, I use three different port numbers for the 3 servers I have at home:

By using non-standard ports, there’s also a side benefit of a little extra security.

In Remote Desktop Client, you would add the port number to the Computer name like this:

Happy Remote Desktoping!

The post Make all of your home servers reachable by Remote Desktop appeared first on Chinh Do.

So I started to look for a way to manage/categorize all of these raw videos and I think I’ve found a pretty good and free solution: Picasa (from Google).

The current version of Picasa for Windows (version 3.8) allows you to specify titles for images and videos, as well as assigning various tags to them. The user interface is very user-friendly. And the actual searches are typical Google-lighting-fast.

What I do is I go through each raw video take, give it a title and assign some tags. Then later I am able to very quickly search for videos. For example, if I need a take involving a dolly pan to the right, I just type “dolly pan” into Picasa and instantly the takes with those keywords come up.

The post Managing, Tagging, Organizing Videos with Picasa appeared first on Chinh Do.

Wow! 25 World cup soccer games in 3D! Wait a minute, I think I see some dead pixels on my current HDTV.

The post World Cup 2010 In Glorious 3D appeared first on Chinh Do.

• Instant messaging is not necessarily one-one-one. At any given moment, one person may be engaged in several simultaneous instant messaging conversations.
• Even though your messages will be displayed on the recipient’s screen immediately, the recipient may not be able to read the messages and respond immediately.
• With most instant messaging applications, the recipient cannot read your messages until you press the Enter key to send it. This limitation inherently makes instant messaging significantly slower than voice conversations.
• Most people can speak faster than they can type.

Here are some guidelines on basic/everyday instant messaging etiquette that will help you and your co-workers make the most out of this communication medium.

Include Relevant Info/Questions in The First Message

Avoid typing greetings or non-essential messages separately first. Include the question or relevant information in your first message.

Not-so-good examples (avoid this)

Conversation 1

• Jane (10:54:50 AM): Hi
• Mike (10:55:01 AM): Hello
• Jane (10:55:06 AM): The test server will be restarted in 1 minute.
• Mike (10:56:11 AM): Thanks for the info

Conversation 2

• Jane (2:15:08 PM): Good afternoon
• Mike (2:15:13 PM): Good afternoon
• Jane (2:15:13 PM): Do you have a few minutes to talk on the phone re project A?
• Mike (2:15:18 PM): Sure, let me call you.

Conversation 3

• Jane (4:03:30 PM): You there?
• Mike (4:03:35 PM): Yes
• Jane (4:03:42 PM): Can we have a short team meeting in conf room A
• Mike (4:03:50 PM): Sure. Be there in 5.

Good examples

Conversation 1

• Jane (10:54:50 AM): Hi, the test server will be restarted in 1 minute.
• Mike (10:54:55 AM): Thanks for the info

Conversation 2

• Jane (2:15:08 PM): Good afternoon, do you have a few minutes to talk on the phone re project A?
• Mike (2:15:13 PM): Sure I will call you.

Conversation 3

• Jane (4:03:30 PM): Can we have a short team meeting in conf room A?
• Mike (4:03:35 PM): Sure, be right there.

Every time you send a message, the recipient is disrupted from whatever he/she is doing. The more you can delay this disruption the better, even if it’s only seconds. It’s also not necessary to ask the recipient if he/she is there. That’s what the online status (away/available) is for. You can just type your message or ask your question. One exception to this would be if your message contains sensitive information.

Send Complete Messages

Good:

• Jane (9:15:23 AM): Hi, all integration tests are failing in the integration environment for the admin user group. Can you take a look?”

No so good:

• Jane (9:15:23 AM): Hi, all integration tests are failing…
• Jane (9:15:27 AM): the integration environment…
• Jane (9:15:32 AM): for the admin user group…”
• Jane (9:15:38 AM): Can you take a look?

If you have to send several sentences in sequence, compose them in a separate editor first (notepad would do), then type them out quickly in succession. That way the recipient does not have to wait for you while you type your next message, make editing corrections, etc.

Do Not Expect Responses Immediately

Unless the recipient reports to you, do not expect a response immediately after you type a message. The recipient may be in the middle of five other instant message conversations, on the phone, or working on something else more important. If the recipient’s status is not set to “Away”, and you have not received a response after a few minutes, it’s ok to ping him/her again. Maybe they forgot about your message.

Update Your Status and Respect Others’ Status

If you need to step out, change your status to “Away”, or “In a Meeting”, or “Back in an Hour”, or whatever is appropriate for the situation. This tells everyone else that you are not available to respond to messages immediately and save them from having to wait for your responses.

Pick Up the Phone

If the instant messaging conversation starts to go into lots of details and may make take longer than a few minutes, consider picking up the phone and continue the conversation there.

The post Instant Messaging Etiquette for the Workplace appeared first on Chinh Do.

Yes, one would think that being able to quickly identify any running window would be a basic feature of any modern so called window operation system. It’s 25 years after the first release of Microsoft Windows, and the sad truth is that you still often cannot easily identify running windows.

Look at the example below. If you are not familiar with this utility, and you came back to your laptop seeing this, would you know what it’s about? Should you click Yes or No? Is this a legitimate application, or something more sinister?

The first obvious problem is the missing message. That’s forgivable however. Bugs happen, files get corrupted, language resource files go missing, etc. What’s not acceptable is for the Windows OS not to provide any method to identify misbehaving windows.

So how about it Microsoft? Let us easily find out identifying information about any running Windows. Perhaps with with a click of a button, we can see:

• Name of owning application/process
• Name of vendor (if available)
• Folder where executable resides
• Date the application was installed
• User who installed the application
• If the user didn’t run the application himself, identify the parent process or service that launches the application (shortcut in Startup folder, registry, etc.)
• Available code signatures

For now, if you want to identify any visible window, use Sysinternals’ Process Explorer. Drag the “Find Window’s Process” icon and drop it on top of the target window and Process Explorer will highlight the owner process in its window. From there, you can get the executable name, company name, folder location, etc.

The post Show Your Papers! appeared first on Chinh Do.

So, the first thing I thought of is a remote control for the thermostat. Well, no surprise, they do make them. Apparently, my problem is fairly common for two-story homes with a single HVAC system. This Lux TX9000RF Programmable Thermostat with Remote looked very promising to me. A product like this would allow adjusting the thermostat temperature from anywhere in the house.

That still requires some work however. Hmm… what if there is a thermostat that can read the current temperature from a remote sensor? Bingo: they make those too. There are not many to be found, and after searching around, I decided to go for the  Honeywell YTHX9321R5003 Prestige HD Thermostat Kit and I’ve been very happy with the result so far. This kit is expensive, but very well made and it works as advertised. It also looks very nice. The kit includes the thermostat, a remote control/sensor, and an outdoor sensor. This kit is in Honeywell Pro Install line, which means it’s sold mainly through HVAC contractors and installers. I found the installation process only slightly more complicated than a regular programmable thermostat. The only thing you need to watch out for is that this thermostat requires a 24vac Common wire (commonly black in color), which may not be available in your setup. If that is the case, then you will need to run/fish a new wire from your furnace – a pretty big job.

Now with this cool new gadget hooked up and everything humming, all I have to do is bring the remote with me to the bedroom and push the button on it named “Read temp from this device” and I am set for the night. If I ever want to tweak the temperature for some reason, I can do it right there with the remote. If only everything else was this easy!

The post Taking Control of Your Thermostat appeared first on Chinh Do.