Using jq to fetch key value from json output

In this post, you will get insights into using jq tool. For parsing json data & retrieve specific fields. Which can be further processed or used for any other operations. To know more about jq follow the link

Objective: Parse the json from a file or end-point using `jq` and retrieve id-value pair for further process.

Additional condition: if the field `name` have pattern: “test1, test2, test3…testN” then create a CSV file with fields: id,name,value.

Example Data Set: sample-output.json

Let’s export the sample data in a variable.

export dataseturl=https://gist.githubusercontent.com/akoserwal/4817cccbb53262056405bcc1eef37443/raw/7af7a0a6a60b5e2b8699f0c6e779c236e8f1055d/sample-output.json

Try to use the Jq with Array/Object Value Iterator: .[]

curl $dataseturl | jq -r '.[]'

Output

{
  "id": "3424234",
  "name": "ex2",
  "value": "exssasda"
}
{
  "id": "342423e4",
  "name": "test3",
  "value": "tyetryxssasda"
}
{
  "id": "5453424234",
  "name": "ex1",
  "value": "t55exssasda"
}
{
  "id": "4353424234",
  "name": "test8",
  "value": "fdgexssasda"
}
{
  "id": "3435424234",
  "name": "test9",
  "value": "exssasda"
}

If we add Object Identifier-Index like: ‘.id & .value’ to previous command

curl $dataseturl | jq -r '.[].id,.[].value'
  
3424234
342423e4
5453424234
4353424234
3435424234
exssasda
tyetryxssasda
t55exssasda
fdgexssasda
exssasda

In the above output we can see we are getting all the elements matching key “id” and then “value”. But our objective is get the key value pair like id:value.

Solution

To break it down, we are using above mentioned filters but with help of pipe operator. Sending output from one to another and finally converting into csv format (command separated).

`Array/Object Value Iterator | Object Identifier-Index | @csv`

It will look like this

curl $dataseturl | jq -r '.[] | [.id, .name, .value] | @csv'
"3424234","ex2","exssasda"
"342423e4","test3","tyetryxssasda"
"5453424234","ex1","t55exssasda"
"4353424234","test8","fdgexssasda"
"3435424234","test9","exssasda"

To delete the Quotation Marks (“) using linux tool: tr

curl $dataseturl | jq -r ‘.[] | [.id, .name, .value] | @csv’ | tr -d ‘“‘
3424234,ex2,exssasda
342423e4,test3,tyetryxssasda
5453424234,ex1,t55exssasda
4353424234,test8,fdgexssasda
3435424234,test9,exssasda

Now we have the desired output format. But our goal is to further process the output line by line and do some conditional check.

Leveraging Internal Field Separator (IFS): IFS is a special shell variable. It is used to separate a pattern into tokens for some operations.The value of IFS, typically includes the space, tab, and the newline.

while IFS=, read -r id name value ; do
    echo $id:$value
done

Finally we get the desired result

Key value pair output id:value which can be used for making further API calls or do some processing.

3424234:exssasda
342423e4:tyetryxssasda
5453424234:t55exssasda
4353424234:fdgexssasda
3435424234:exssasda

We have defined a condition in the beginning.

if the field `name` have pattern: “test1, test2, test3…testN” then create a CSV file with fields: id,name,value.

To achieve this lets add a if-else statement in the while loop.

if  [[ $name =~ test* ]];
then
echo "$id,$name,$value" >> generated.csv
fi

Now, if we run entire the solution script

You will see a file generated.csv with output

342423e4,test3,tyetryxssasda
4353424234,test8,fdgexssasda
3435424234,test9,exssasda
342423e4,test3,tyetryxssasda
4353424234,test8,fdgexssasda
3435424234,test9,exssasda
342423e4,test3,tyetryxssasda
4353424234,test8,fdgexssasda
3435424234,test9,exssasda
342423e4,test3,tyetryxssasda
4353424234,test8,fdgexssasda
3435424234,test9,exssasda

Conclusion

In this post, we covered parsing json data with powerful tool jq. Which allowed us to parse a nested json object array & retrieve key-value pair. Used filters like Array/Object Value Iterator, Object Identifier-Index & convert into CSV format. Finally with help of IFS, converted into individual tokens which can be further processed on some condition or used for making further calls to new end-point.

Hope you find this post useful. Please give a cheer if you like this post.

Thank you so much.