bookmark_borderParse JSON Responses using Bash Scripts

I was working with the categorify.org site and I wanted to parse through the API’s response. The response was in JSON format.

There are a number of different ways you can do this, but if you want a quick, simple, way that uses existing tools you probably already have installed, this is for you.

To parse through the JSON response I used Python.

Here is an example of what I was doing:

curl -s https://categorify.org/api?website=pornhub.com

If you do this, the response is something like this:

$ curl -s https://categorify.org/api?website=pornhub.com

{“domain”:”pornhub.com”,”ip”:”31.192.120.36″,”country-code”:”NL”,”country”:”Netherlands”,”rating”:{“language”:true,”violence”:false,”nudity”:true,”adult”:true,”value”:”R & NSFW”,”description”:”Adult-only content and not safe for a work environment”},”category”:[“Adult\/Pornography”],”keyword_heatmap”:{“videos”:99,”free”:85,”pornhub”:61,”porn”:55,”models”:54,”premium”:45,”content”:45,”rated”:44,”photos”:44,”exclusive”:43,”party”:43,”gifs”:42,”sister”:39,”discover”:36,”online”:36,”video”:35,”albums”:34,”pornhubcom”:33,”cancel”:33,”subscribed”:30}}

What I specifically wanted was the category value: “category”:[“Adult\/Pornography”]

So I modified my command to pipe my output to python and ran the following:

$ curl -s https://categorify.org/api?website=pornhub.com | python -c ‘import json,sys;obj=json.load(sys.stdin);print obj[“category”]’

The section you’re most interested in is this: print obj[“category”]

You have to choose which object you want to print. In my case, I wanted the category object as it holds the value I am looking for.

You can use this to parse any JSON response, just update the print object with one that corresponds to your results. The beautiful part is that this should be available on most major OS platforms without additional installations.


Side note, here are two variations depending on which version of Python you have on your machine.

Python 3

python3 -c "import sys, json; print(json.load(sys.stdin)['category'])"

Python 2

python2 -c "import sys, json; print json.load(sys.stdin)['category']"

Site note, to the side note, if you simply want a pretty display of the json you can pipe the output into “| python3 -m json.tool“. This will give you something much easier to read:

$ curl -s https://categorify.org/api?website=pornhub.com | python3 -m json.tool                                                         {                                                                                                                                                                     "domain": "pornhub.com",                                                                                                                                          "ip": "31.192.120.36",                                                                                                                                            "country-code": "NL",                                                                                                                                             "country": "Netherlands",                                                                                                                                         "rating": {                                                                                                                                                           "language": true,                                                                                                                                                 "violence": false,                                                                                                                                                "nudity": true,                                                                                                                                                   "adult": true,                                                                                                                                                    "value": "R & NSFW",                                                                                                                                              "description": "Adult-only content and not safe for a work environment"                                                                                       },                                                                                                                                                                "category": [                                                                                                                                                         "Adult/Pornography"                                                                                                                                           ],                                                                                                                                                                "keyword_heatmap": {                                                                                                                                                  "videos": 99,                                                                                                                                                     "free": 85,                                                                                                                                                       "pornhub": 61,                                                                                                                                                    "porn": 55,                                                                                                                                                       "models": 54,                                                                                                                                                     "premium": 45,                                                                                                                                                    "content": 45,                                                                                                                                                    "rated": 44,                                                                                                                                                      "photos": 44,                                                                                                                                                     "exclusive": 43,                                                                                                                                                  "party": 43,                                                                                                                                                      "gifs": 42,                                                                                                                                                       "sister": 39,                                                                                                                                                     "discover": 36,                                                                                                                                                   "online": 36,                                                                                                                                                     "video": 35,                                                                                                                                                      "albums": 34,                                                                                                                                                     "pornhubcom": 33,                                                                                                                                                 "cancel": 33,                                                                                                                                                     "subscribed": 30                                                                                                                                              }                                                                                                                                                             }                           

bookmark_borderPi-Hole Error: Could not update local repository. Contact Support.

When you are running Pi-Hole for the first time, you might run into an error that reads:

Could not update local repository. Contact Support.

This can be for a couple of different reasons, but these are the ones that have been most effective:

  1. Verify that you chose the right interface when installing Pi-Hole. For instance, I chose the ethernet interface on the installation process. When it went to install, it assumed the internet was on this interface. I reran the installer, chose the WiFi interface, and it worked. Alternatively, connect it via the ethernet interface and that too should work.
  2. If you are running this on a new install, or what you believe to be a new install, try running this command:

pi@raspberrypi:~ $ sudo git clone -q –depth 1 https://github.com/pi-hole/pi-hole.git /etc/.pihole

If it returns this, it’s an update problem, not a new install problem:

fatal: destination path ‘/etc/.pihole’ already exists and is not an empty directory.

Good luck!

bookmark_borderUse Bash Script to Monitor The Status of Service

When you manage multiple servers it’s sometimes impossible to stay ahead of the various administrative tasks, which is why automation is so important.

If you’re working on a linux based server and want to monitor the status of a service, here is a quick an easy way to automate that process.

#!/bin/bash
# Script to find if a service is running

for i in ossec-monitord ossec-logcollector ossec-integratord;

 do ps auwx | grep -v grep | grep $i >/dev/null 2>&1 ;

   if [ $? = 0 ];

    then

     echo `date “+%Y-%m-%d %H:%M “`”$i Running…”;

    else

     echo `date “+%Y-%m-%d %H:%M “`”$i not running…”;

fi;

What we did above is create a simple loop looking for three distinct services:

  • ossec-monitord
  • ossec-logcollector
  • ossec-integratord

Those three services are assigned to the i variable, and that variable is then passed into the grep query here:

ps auwx | grep -v grep | grep $i >/dev/null 2>&1 ;

What we’re also doing above is cutting out any grep inquiries, because if you were to run a grep command it will show as a process as shown here in red:

root@server:~/scripts# ps auwx | grep ossec-logcollector

root      1260  0.0  0.0   4876  1784 ?        S    May19   0:03 /var/ossec/bin/ossec-logcollector

root     29353  0.0  0.0  14428  1116 pts/0    S+   20:59   0:00 grep –color=auto ossec-logcollector

By cutting out the grep request you see this response:

root@server:~/scripts# ps auwx | grep -v grep | grep ossec-logcollector

root      1260  0.0  0.0   4876  1784 ?        S    May19   0:03 /var/ossec/bin/ossec-logcollector

This is important because this   if [ $? = 0 ]; is looking for the grep exit value of 0, which states:

EXIT STATUS
    The grep utility exits with one of the following values:

    0     One or more lines were selected.
    1     No lines were selected.
    >1    An error occurred.

With this selection, if you run the grep and there are no service running it would still find the grep service itself. It’d give you a false positive response.

The echo command then prints the status of the service:

echo `date “+%Y-%m-%d %H:%M “`”$i Running…”;

It passes each argument through the loop. If the service is found to be running it prints:

2020-05-19 18:27 ossec-monitord Running…

2020-05-19 18:27 ossec-logcollector Running…

2020-05-19 18:27 ossec-integratord Running…

bookmark_borderTips and Tricks: Blocking DNS requests via Iptables

Iptables has to be one of the tools that I use the most on my day to day work. The default firewall tool chain on Linux has a lot of options to filter pretty much any traffic you wish.

In this Tips and Tricks, we will show you how to block DNS requests (domain names + request types) via iptables. Enjoy!

Continue reading “Tips and Tricks: Blocking DNS requests via Iptables”

bookmark_borderBriefly unavailable for scheduled maintenance. Check back in a minute.

Every now and then, depending on what you’re doing, you might encounter this problem with your WordPress site. I often see it after running an update:

Continue reading “Briefly unavailable for scheduled maintenance. Check back in a minute.”

bookmark_borderCreate MySql Database Backup in Linux Using MariaDB

If you need to do a MySQL database backup in linux, this is the basic command structure you want to use:

# mysqldump -u [username] -p [database name] > [filename]-$(date +%F).sql

This will prompt the password when you hit enter.

Continue reading “Create MySql Database Backup in Linux Using MariaDB”

bookmark_borderInspecting DNS traffic via tcpdump

If you ever wondered what is going on at the DNS level on your computer (or network), tcpdump can be a useful tool for you.

TCPdump basics

Tcpdump is a tool that allows you to inspect any packet (TCP, UDP, etc) and its content as they pass through an interface through the libpcap module. The syntax is very simple, but the basics of the command require the network interface name, the protocol and the restrictions of what you are trying to inspect (more on that later):

Continue reading “Inspecting DNS traffic via tcpdump”

bookmark_borderActive Domain Error in GravityForms MailGun plugin

For whatever reason I have was having the hardest time getting the MailGun plugin for GravityForms to use my From Email domain in WordPress.

Every time I tried to update the settings, I’d get the following error:

Continue reading “Active Domain Error in GravityForms MailGun plugin”

bookmark_borderBlocking HTTP requests via Iptables for a specific domain

In a previous article, we showed how to block specific domains at the DNS level using iptables. Today, we will expand into that and show how to also block HTTP requests for a specific domain (or URL) in there.

Iptables String Matching

Iptables string matching is very powerful and easier to use than the hex-string module we used before. When you specify -m string –string, it will activate the string module and inspect at the packet content for the keyword you are looking for.

Continue reading “Blocking HTTP requests via Iptables for a specific domain”

bookmark_borderBinding multiple IPv6 addresses automatically

Most servers get a IPv6 range (/64) by default. That means that you have millions of IP addresses to use for whatever you feel like. However, assigning them manually to your interfaces can be a bit painful.

Assigning all /64 IPv6 addresses with 1 command

However, there is a trick with the ip route command that allows you to link your /64 to the local interface and cover all of them automatically:

Continue reading “Binding multiple IPv6 addresses automatically”