The .tar.gz and .zip archive formats are quite ubiquitous and with good reason. For decades they have served as the backbone of our data archiving and transfer needs. With the advent of multi-core and multi-socket CPU architectures, little unfortunately has been done to leverage the wider number of processors. While archiving then compressing a directory may seem like the intuitive sequence, we will show how compressing files before adding them to a .
Ranger is one of the few packages I simply can’t live without. Ranger is one of my most used CLI tools and my defacto standard for navigating directories and transferring files. If you love ranger like me though you have come to deal with its many bugs and excentricities. One of the most frustrating for me personally is rangers automatic previewing of archives. While this feature is great if you are dealing with nothing but tiny archives, but not in a directory full of +100GB archives… At that point ranger becomes unusable and a general nightmare.
This script is designed to download, convert, and sort GDELT source files automatically into a user-specified path. https://www.gdeltproject.org/data.html#rawdatafiles
*Note: This script is in BETA and may have bugs. https://github.com/JustinTimperio/gdelt-diff
Description To maintain consistency across my personal datasets, these files are converted upon download from .zip to .gz. Due to the large size of the GDELT source files extra care is taken to ensure that as much file processing as possible is done in ram.