Checks the accessibility of URLs in the clear web and the Tor/Onion network.
Supported operating systems:
Please read the "Getting Started" section for the first steps.
The format of the command line arguments used by the programs are described in the section "Command line arguments format".
Report bugs to werner@gaijin.at
Get the newest version at https://www.gaijin.at
Directories in the archive or image file:
Extract all files and folders in the downloaded archive file to a location of your choice, e.g your personal folder.
Open the downloaded image file. Drag the contained directory from the image file to your home directory, for example, and drop it.
Note: Some directories, such as the desktop, have special security restrictions. In these directories, the programs and example scripts do not work.
The downloaded .dmg image file is not signed. Downloaded applications or containers are marked and cannot be executed without additional user interaction. To avoid having to open and approve each individual program, you can follow these steps:
You can try out the example scripts in the "examples" directory to see the output and in the script code you will find the arguments that are used to execute the program.
For detailed help on a program, run it in the shell or console with the option
-h or --help, e.g. application.exe --help under Windows or
./application --help under MacOS.
Command line arguments are separated with a space. Arguments with spaces must enclosed with quotes (depending on the operating system and situation with single or double quotes).
Short options starts with a single hyphen and can be grouped. This means that
the arguments -a -b -c are the same as -abc. This works only for options
without a parameter. Long options starts with two hyphens.
Both option types can have additionally parameters (values). The parameters
can be written as own arguments, directly after the option argument like
-o PARAM1 PARAM2.
A single parameter can be given in the same argument as the option, separated
by a colon (:) or an equals sign (=) like -o:PARAM1 or -o=PARAM1.
Multiple parameters can be given as option arguments in the form
-o=PARAM1 -o=PARAM2 or --option=PARAM1 --option=PARAM2.
Non-option arguments usually contain data such as a file or directory path.
The "Usage" section of the program help (-h/--help) provides a brief overview of the possible options and non-options arguments and whether they are optional or required.
An argument in square brackets such as [option] is optional, without square
brackets, the argument is required. A pipe character (vertical bar) will be
used for required arguments, which can contain several different options or
arguments. Example: -u=URL | FILE means that you must set the option "u"
or give a file as non-option argument.
Checks the accessibility of URLs in the clear web and the Tor/Onion network.
URLReport can output the result of the check as a single file in the formats TSV, CSV and HTML, as well as a comprehensive HTML report.
The data fields and columns of the simple output can be customized. This output is particularly suitable for a quick overview of the reachable URLs or for further processing of the data with another program.
The HTML report (URL report) allows you to view the checked and accessible websites as a secure HTML version and as plain text. The technical details, such as the headers or redirects, can also be viewed.
In the folder "examples" contains several Shell scripts respectively Windows batch files that demonstrate different output options. The output files will be written to the subfolder "files". (The file name extensions of the scripts differ depending on the platform.)
"urlreport_csv" and "urlreport_html" output the results of URLs from the clear-web as CSV resp. HTML files.
"urlreport_report" and "urlreport_report_tor" generate a HTML report.
In the script "urlreport_report_tor" all options are set to values that are suitable for .onion URLs, especially with regard to the timeout and the number of connection attempts.
The "gui" folder contains two shell scripts or batch files for URLReport. "urlreport" displays a GUI for exporting the data to a file. The script "urlreport_report_tor" displays a GUI specifically for checking .onion URLs via Tor and exporting them as a complete HTML report.
For requests via Tor a timeout of at least 30 seconds and multiple connection attempts are recommended.
Extracts .onion URLs from plain text files.
URLs can be excluded with regular expressions, e.g. to ignore URLs with CSS or image files. Duplicates can be ignored and not printed. The domain name without schema, path and parameters, can be output in addition to the URL itself.
Note: The number of found and exported URLs will be shown at the end of each file and at the end in total - except the option -s/--silent is set.
If the option -d/dnadd is set, the number of exported URLs can be greater than the URLs found, because the added domain name counts too.
Onion URL Extractor can be used to read and process exported search hits from X-Ways Forensics.
[a-z2-7]{56}\.onion
to find all .onion domains.Now the file can be processed with the script "onionextract", which is already contained in the XWF folder.
The extracted URLs will be stored in the file "urls.txt". Additionally, the domain name of each URL will be added and all URLs will be unique in the output file.
To be able to check the .onion URLs, the "tor" program must be executed. To do this under Windows, start the program "Tor Browser\Browser\TorBrowser\Tor\tor.exe" in the Tor installation or under MacOS the program "/Applications/Tor Browser.app/Contents/MacOS/Tor/tor".
Note: The program "tor" is not the same as the "Tor Browser".
To run the program more easily, you can, for example, create a shortcut and store it in the "XWF" folder.
The extracted URLs can then be checked with the script "urlreport". The output will be written to the directory "URLReport\XWF\URL Report".