bookmark_borderParse JSON Responses using Bash Scripts

I was working with the categorify.org site and I wanted to parse through the API’s response. The response was in JSON format.

There are a number of different ways you can do this, but if you want a quick, simple, way that uses existing tools you probably already have installed, this is for you.

To parse through the JSON response I used Python.

Here is an example of what I was doing:

curl -s https://categorify.org/api?website=pornhub.com

If you do this, the response is something like this:

$ curl -s https://categorify.org/api?website=pornhub.com

{“domain”:”pornhub.com”,”ip”:”31.192.120.36″,”country-code”:”NL”,”country”:”Netherlands”,”rating”:{“language”:true,”violence”:false,”nudity”:true,”adult”:true,”value”:”R & NSFW”,”description”:”Adult-only content and not safe for a work environment”},”category”:[“Adult\/Pornography”],”keyword_heatmap”:{“videos”:99,”free”:85,”pornhub”:61,”porn”:55,”models”:54,”premium”:45,”content”:45,”rated”:44,”photos”:44,”exclusive”:43,”party”:43,”gifs”:42,”sister”:39,”discover”:36,”online”:36,”video”:35,”albums”:34,”pornhubcom”:33,”cancel”:33,”subscribed”:30}}

What I specifically wanted was the category value: “category”:[“Adult\/Pornography”]

So I modified my command to pipe my output to python and ran the following:

$ curl -s https://categorify.org/api?website=pornhub.com | python -c ‘import json,sys;obj=json.load(sys.stdin);print obj[“category”]’

The section you’re most interested in is this: print obj[“category”]

You have to choose which object you want to print. In my case, I wanted the category object as it holds the value I am looking for.

You can use this to parse any JSON response, just update the print object with one that corresponds to your results. The beautiful part is that this should be available on most major OS platforms without additional installations.


Side note, here are two variations depending on which version of Python you have on your machine.

Python 3

python3 -c "import sys, json; print(json.load(sys.stdin)['category'])"

Python 2

python2 -c "import sys, json; print json.load(sys.stdin)['category']"

Site note, to the side note, if you simply want a pretty display of the json you can pipe the output into “| python3 -m json.tool“. This will give you something much easier to read:

$ curl -s https://categorify.org/api?website=pornhub.com | python3 -m json.tool                                                         {                                                                                                                                                                     "domain": "pornhub.com",                                                                                                                                          "ip": "31.192.120.36",                                                                                                                                            "country-code": "NL",                                                                                                                                             "country": "Netherlands",                                                                                                                                         "rating": {                                                                                                                                                           "language": true,                                                                                                                                                 "violence": false,                                                                                                                                                "nudity": true,                                                                                                                                                   "adult": true,                                                                                                                                                    "value": "R & NSFW",                                                                                                                                              "description": "Adult-only content and not safe for a work environment"                                                                                       },                                                                                                                                                                "category": [                                                                                                                                                         "Adult/Pornography"                                                                                                                                           ],                                                                                                                                                                "keyword_heatmap": {                                                                                                                                                  "videos": 99,                                                                                                                                                     "free": 85,                                                                                                                                                       "pornhub": 61,                                                                                                                                                    "porn": 55,                                                                                                                                                       "models": 54,                                                                                                                                                     "premium": 45,                                                                                                                                                    "content": 45,                                                                                                                                                    "rated": 44,                                                                                                                                                      "photos": 44,                                                                                                                                                     "exclusive": 43,                                                                                                                                                  "party": 43,                                                                                                                                                      "gifs": 42,                                                                                                                                                       "sister": 39,                                                                                                                                                     "discover": 36,                                                                                                                                                   "online": 36,                                                                                                                                                     "video": 35,                                                                                                                                                      "albums": 34,                                                                                                                                                     "pornhubcom": 33,                                                                                                                                                 "cancel": 33,                                                                                                                                                     "subscribed": 30                                                                                                                                              }                                                                                                                                                             }                           

bookmark_borderPi-Hole Error: Could not update local repository. Contact Support.

When you are running Pi-Hole for the first time, you might run into an error that reads:

Could not update local repository. Contact Support.

This can be for a couple of different reasons, but these are the ones that have been most effective:

  1. Verify that you chose the right interface when installing Pi-Hole. For instance, I chose the ethernet interface on the installation process. When it went to install, it assumed the internet was on this interface. I reran the installer, chose the WiFi interface, and it worked. Alternatively, connect it via the ethernet interface and that too should work.
  2. If you are running this on a new install, or what you believe to be a new install, try running this command:

pi@raspberrypi:~ $ sudo git clone -q –depth 1 https://github.com/pi-hole/pi-hole.git /etc/.pihole

If it returns this, it’s an update problem, not a new install problem:

fatal: destination path ‘/etc/.pihole’ already exists and is not an empty directory.

Good luck!

bookmark_borderUse Bash Script to Monitor The Status of Service

When you manage multiple servers it’s sometimes impossible to stay ahead of the various administrative tasks, which is why automation is so important.

If you’re working on a linux based server and want to monitor the status of a service, here is a quick an easy way to automate that process.

#!/bin/bash
# Script to find if a service is running

for i in ossec-monitord ossec-logcollector ossec-integratord;

 do ps auwx | grep -v grep | grep $i >/dev/null 2>&1 ;

   if [ $? = 0 ];

    then

     echo `date “+%Y-%m-%d %H:%M “`”$i Running…”;

    else

     echo `date “+%Y-%m-%d %H:%M “`”$i not running…”;

fi;

What we did above is create a simple loop looking for three distinct services:

  • ossec-monitord
  • ossec-logcollector
  • ossec-integratord

Those three services are assigned to the i variable, and that variable is then passed into the grep query here:

ps auwx | grep -v grep | grep $i >/dev/null 2>&1 ;

What we’re also doing above is cutting out any grep inquiries, because if you were to run a grep command it will show as a process as shown here in red:

root@server:~/scripts# ps auwx | grep ossec-logcollector

root      1260  0.0  0.0   4876  1784 ?        S    May19   0:03 /var/ossec/bin/ossec-logcollector

root     29353  0.0  0.0  14428  1116 pts/0    S+   20:59   0:00 grep –color=auto ossec-logcollector

By cutting out the grep request you see this response:

root@server:~/scripts# ps auwx | grep -v grep | grep ossec-logcollector

root      1260  0.0  0.0   4876  1784 ?        S    May19   0:03 /var/ossec/bin/ossec-logcollector

This is important because this   if [ $? = 0 ]; is looking for the grep exit value of 0, which states:

EXIT STATUS
    The grep utility exits with one of the following values:

    0     One or more lines were selected.
    1     No lines were selected.
    >1    An error occurred.

With this selection, if you run the grep and there are no service running it would still find the grep service itself. It’d give you a false positive response.

The echo command then prints the status of the service:

echo `date “+%Y-%m-%d %H:%M “`”$i Running…”;

It passes each argument through the loop. If the service is found to be running it prints:

2020-05-19 18:27 ossec-monitord Running…

2020-05-19 18:27 ossec-logcollector Running…

2020-05-19 18:27 ossec-integratord Running…

bookmark_borderBriefly unavailable for scheduled maintenance. Check back in a minute.

Every now and then, depending on what you’re doing, you might encounter this problem with your WordPress site. I often see it after running an update:

Continue reading “Briefly unavailable for scheduled maintenance. Check back in a minute.”

bookmark_borderCreate MySql Database Backup in Linux Using MariaDB

If you need to do a MySQL database backup in linux, this is the basic command structure you want to use:

# mysqldump -u [username] -p [database name] > [filename]-$(date +%F).sql

This will prompt the password when you hit enter.

Continue reading “Create MySql Database Backup in Linux Using MariaDB”

bookmark_borderActive Domain Error in GravityForms MailGun plugin

For whatever reason I have was having the hardest time getting the MailGun plugin for GravityForms to use my From Email domain in WordPress.

Every time I tried to update the settings, I’d get the following error:

Continue reading “Active Domain Error in GravityForms MailGun plugin”

bookmark_borderMySQL – ERROR 1819 (HY000): Your password does not satisfy the current policy requirements

If you run into this error it’s a strong indicator that the Password Validation Plugin is installed.

The quick and dirty way to fix this is to uninstall the plugin. You will need to be the root user in the database.

Continue reading “MySQL – ERROR 1819 (HY000): Your password does not satisfy the current policy requirements”

bookmark_borderRemoving Index.php From the URL Slug in WordPress

This post assumes you’re having a hard time removing the index.php file name from the URL slug of your WordPress site.

Looks like something like this:

defragged.org/index.php/%postname%

This means one of two things:

Continue reading “Removing Index.php From the URL Slug in WordPress”

bookmark_borderLets Encrypt: Unable to install the certificate

You’ve heard it’s important to install Let’s Encrypt (LE). You spin up your Ubuntu 18.04 machine and try to use https://certbot.eff.org/.

You run the command:

certbot –apache -d domain.com

You are greeted with:

Continue reading “Lets Encrypt: Unable to install the certificate”

bookmark_borderHow To List UFW Rules When The Application is Inactive or Disabled

When working on your server you might need to disable the Uncomplicated Firewall (UFW). When doing so you’ll notice it doesn’t display your rules when the application is inactive.

Continue reading “How To List UFW Rules When The Application is Inactive or Disabled”