Script 'mail_helper' called by obssrc Hello community, here is the log from the commit of package urlscan for openSUSE:Factory checked in at 2021-11-02 19:19:14 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Comparing /work/SRC/openSUSE:Factory/urlscan (Old) and /work/SRC/openSUSE:Factory/.urlscan.new.1890 (New) ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Package is "urlscan" Tue Nov 2 19:19:14 2021 rev:9 rq:928790 version:0.9.7 Changes: -------- --- /work/SRC/openSUSE:Factory/urlscan/urlscan.changes 2021-05-17 18:45:22.416608611 +0200 +++ /work/SRC/openSUSE:Factory/.urlscan.new.1890/urlscan.changes 2021-11-02 19:19:53.927117754 +0100 @@ -1,0 +2,9 @@ +Fri Oct 8 14:30:28 UTC 2021 - Paolo Stivanin <[email protected]> + +- update to 0.9.7: + * User-option -W --whitespace-off: remove blank lines and ellipses + * Update documentation for feature --whitespace-off + * Correct regex example in urlscan.1 + * Sort and align docstrings + +------------------------------------------------------------------- Old: ---- urlscan-0.9.6.tar.gz New: ---- urlscan-0.9.7.tar.gz ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Other differences: ------------------ ++++++ urlscan.spec ++++++ --- /var/tmp/diff_new_pack.ZhJjAg/_old 2021-11-02 19:19:54.355118082 +0100 +++ /var/tmp/diff_new_pack.ZhJjAg/_new 2021-11-02 19:19:54.359118085 +0100 @@ -18,7 +18,7 @@ %define python_flavor python3 Name: urlscan -Version: 0.9.6 +Version: 0.9.7 Release: 0 Summary: An other URL extractor/viewer License: GPL-2.0-or-later @@ -56,10 +56,10 @@ %files %license COPYING -%doc README.rst %{_bindir}/%{name} %{python_sitelib}/%{name} %{_mandir}/man1/%{name}.1%{?ext_man} %doc %{_defaultdocdir}/%{name}/muttrc +%dir %{_docdir}/urlscan %changelog ++++++ urlscan-0.9.6.tar.gz -> urlscan-0.9.7.tar.gz ++++++ diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/urlscan-0.9.6/README.md new/urlscan-0.9.7/README.md --- old/urlscan-0.9.6/README.md 1970-01-01 01:00:00.000000000 +0100 +++ new/urlscan-0.9.7/README.md 2021-10-06 20:10:59.000000000 +0200 @@ -0,0 +1,192 @@ +# Urlscan + +## Contributors + +Scott Hansen \<[email protected]\> (Author and Maintainer) + +Maxime Chatelle \<[email protected]\> (Debian Maintainer) + +Daniel Burrows \<[email protected]\> (Original Author) + +## Purpose and Requirements + +Urlscan is a small program that is designed to integrate with the "mutt" +mailreader to allow you to easily launch a Web browser for URLs contained in +email messages. It is a replacement for the "urlview" program. + +*NOTE* The last version that is Python 2 compatible is 0.9.3. + +Requires: Python 3.6+ and the python-urwid library + +## Features + +Urlscan parses an email message or file and scans it for URLs and email +addresses. It then displays the URLs and their context within the message, and +allows you to choose one or more URLs to send to your Web browser. +Alternatively, it send a list of all URLs to stdout. + +Relative to urlview, urlscan has the following additional features: + +- Support for emails in quoted-printable and base64 encodings. No more stripping + out =40D from URLs by hand! + +- The context of each URL is provided along with the URL. For HTML mails, a + crude parser is used to render the HTML into text. Context view can be toggled + on/off with `c`. + +- URLs are shortened by default to fit on one line. Viewing full URL (for one or + all) is toggled with `s` or `S`. + +- Jump to a URL by typing the number. + +- Incremental case-insensitive search with `/`. + +- Execute an arbitrary function (for example, copy URL to clipboard) instead of + opening URL in a browser. + +- Use `l` to cycle through whether URLs are opened using the Python webbrowser + module (default), xdg-open (if installed) or opened by a function passed on + the command line with `--run` or `--run-safe`. + +- Configure colors and keybindings via ~/.config/urlscan/config.json. Generate + default config file for editing by running `urlscan -g`. Cycle through + available palettes with `p`. Set display width with `--width`. + +- Copy URL to clipboard with `C` or to primary selection with `P`. Requires + xsel or xclip. + +- Run a command with the selected URL as the argument or pipe the selected + URL to a command. + +- Show complete help menu with `F1`. Hide header on startup with `--nohelp`. + +- Use a custom regular expression with `-E` for matching urls or any + other pattern. In junction with `-r`, this effectively turns urlscan + into a general purpose CLI selector-type utility. + +- Scan certain email headers for URLs. Currently `Link`, `Archived-At` and + `List-*` are scanned when `--headers` is passed. + +## Installation and setup + +To install urlscan, install from your distribution repositories (Archlinux), +from Pypi, or do a local development install with pip -e: + + pip install --user urlscan + + OR + + cd <path/to/urlscan> && pip install --user -e . + +**NOTE** + + The minimum required version of urwid is 1.2.1. + +Once urlscan is installed, add the following lines to your .muttrc: + + macro index,pager \\cb "<pipe-message> urlscan<Enter>" "call urlscan to + extract URLs out of a message" + + macro attach,compose \\cb "<pipe-entry> urlscan<Enter>" "call urlscan to + extract URLs out of a message" + +Once this is done, Control-b while reading mail in mutt will automatically +invoke urlscan on the message. + +To choose a particular browser, set the environment variable BROWSER. If BROWSER +is not set, xdg-open will control which browser is used, if it's available.: + + export BROWSER=/usr/bin/epiphany + + +## Command Line usage + + urlscan OPTIONS <file> + + OPTIONS [-c, --compact] + [-d, --dedupe] + [-E, --regex <expression>] + [-f, --run-safe <expression>] + [-g, --genconf] + [-H, --nohelp] + [ --headers] + [-n, --no-browser] + [-p, --pipe] + [-r, --run <expression>] + [-R, --reverse] + [-s, --single] + [-w, --width] + [-W --whitespace-off] + +Urlscan can extract URLs and email addresses from emails or any text file. +Calling with no flags will start the curses browser. Calling with '-n' will just +output a list of URLs/email addressess to stdout. The '-c' flag removes the +context from around the URLs in the curses browser, and the '-d' flag removes +duplicate URLs. The '-R' flag reverses the displayed order of URLs and context. +Files can also be piped to urlscan using normal shell pipe mechanisms: `cat +<something> | urlscan` or `urlscan < <something>`. The '-W' flag condenses the +display output by suppressing blank lines and ellipses lines. + +Instead of opening a web browser, the selected URL can be passed as the argument +to a command using `--run-safe "<command> {}"` or `--run "<command> {}"`. Note +the use of `{}` in the command string to denote the selected URL. Alternatively, +the URL can be piped to the command using `--run-safe <command> --pipe` (or +`--run`). Using --run-safe with --pipe is preferred if the command supports it, +as it is marginally more secure and tolerant of special characters in the URL. + +## Theming + +Run `urlscan -g` to generate ~/.config/urlscan/config.json with the default +color and black & white palettes. This can be edited or added to, as desired. +The first palette in the list will be the default. Configure the palettes +according to the [Urwid display attributes][1]. + +Display width can be set with `--width`. + +## Keybindings + +Run `urlscan -g` to generate ~/.config/urlscan/config.json. All of the keys will +be listed. You can either leave in place or delete any that will not be altered. + +To unset a binding, set it equal to "". For example: `"P": ""` + +The follow actions are supported: + +- `all_escape` -- toggle unescape all URLs (default: `u`) +- `all_shorten` -- toggle shorten all URLs (default: `S`) +- `bottom` -- move cursor to last item (default: `G`) +- `clear_screen` -- redraw screen (default: `Ctrl-l`) +- `clipboard` -- copy highlighted URL to clipboard using xsel/xclip (default: `C`) +- `clipboard_pri` -- copy highlighted URL to primary selection using xsel/xclip (default: `P`) +- `context` -- show/hide context (default: `c`) +- `down` -- cursor down (default: `j`) +- `help_menu` -- show/hide help menu (default: `F1`) +- `link_handler` -- cycle link handling (webbrowser, xdg-open, --run-safe or --run) (default: `l`) +- `open_url` -- open selected URL (default: `space` or `enter`) +- `palette` -- cycle through palettes (default: `p`) +- `quit` -- quit (default: `q` or `Q`) +- `reverse` -- reverse display order (default: `R`) +- `shorten` -- toggle shorten highlighted URL (default: `s`) +- `top` -- move to first list item (default: `g`) +- `up` -- cursor up (default: `k`) + +## Update TLD list (for developers, not users) + +`wget https://data.iana.org/TLD/tlds-alpha-by-domain.txt` + +## Known bugs and limitations + +- Running urlscan sometimes "messes up" the terminal background. This seems to + be an urwid bug, but I haven't tracked down just what's going on. + +- Extraction of context from HTML messages leaves something to be desired. + Probably the ideal solution would be to extract context on a word basis rather + than on a paragraph basis. + +- The HTML message handling is a bit kludgy in general. + +- multipart/alternative sections are handled by descending into all the + sub-parts, rather than just picking one, which may lead to URLs and context + appearing twice. (Bypass this by selecting the '--dedupe' option) + +[1]: http://urwid.org/manual/displayattributes.html#display-attributes "Urwid display attributes" diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/urlscan-0.9.6/README.rst new/urlscan-0.9.7/README.rst --- old/urlscan-0.9.6/README.rst 2021-03-23 06:00:17.000000000 +0100 +++ new/urlscan-0.9.7/README.rst 1970-01-01 01:00:00.000000000 +0100 @@ -1,188 +0,0 @@ -Urlscan -======= - -Contributors ------------- - -Scott Hansen <[email protected]> (Author and Maintainer) - -Maxime Chatelle <[email protected]> (Debian Maintainer) - -Daniel Burrows <[email protected]> (Original Author) - -Purpose and Requirements ------------------------- - -Urlscan is a small program that is designed to integrate with the "mutt" -mailreader to allow you to easily launch a Web browser for URLs contained in -email messages. It is a replacement for the "urlview" program. - -*NOTE* The last version that is Python 2 compatible is 0.9.3. - -Requires: Python 3.6+ and the python-urwid library - -Features --------- - -Urlscan parses an email message or file and scans it for URLs and email -addresses. It then displays the URLs and their context within the message, and -allows you to choose one or more URLs to send to your Web browser. -Alternatively, it send a list of all URLs to stdout. - -Relative to urlview, urlscan has the following additional features: - -- Support for emails in quoted-printable and base64 encodings. No more stripping - out =40D from URLs by hand! - -- The context of each URL is provided along with the URL. For HTML mails, a - crude parser is used to render the HTML into text. Context view can be toggled - on/off with `c`. - -- URLs are shortened by default to fit on one line. Viewing full URL (for one or - all) is toggled with `s` or `S`. - -- Jump to a URL by typing the number. - -- Incremental case-insensitive search with `/`. - -- Execute an arbitrary function (for example, copy URL to clipboard) instead of - opening URL in a browser. - -- Use `l` to cycle through whether URLs are opened using the Python webbrowser - module (default), xdg-open (if installed) or opened by a function passed on - the command line with `--run` or `--run-safe`. - -- Configure colors and keybindings via ~/.config/urlscan/config.json. Generate - default config file for editing by running `urlscan -g`. Cycle through - available palettes with `p`. Set display width with `--width`. - -- Copy URL to clipboard with `C` or to primary selection with `P`. Requires - xsel or xclip. - -- Run a command with the selected URL as the argument or pipe the selected - URL to a command. - -- Show complete help menu with `F1`. Hide header on startup with `--nohelp`. - -- Use a custom regular expression with `-E` for matching urls or any - other pattern. In junction with `-r`, this effectively turns urlscan - into a general purpose CLI selector-type utility. - -- Scan certain email headers for URLs. Currently `Link`, `Archived-At` and - `List-*` are scanned when `--headers` is passed. - -Installation and setup ----------------------- - -To install urlscan, install from your distribution repositories (Archlinux), -from Pypi, or do a local development install with pip -e:: - - pip install --user urlscan - - OR - - cd <path/to/urlscan> && pip install --user -e . - -.. NOTE:: - - The minimum required version of urwid is 1.2.1. - -Once urlscan is installed, add the following lines to your .muttrc: - - macro index,pager \\cb "<pipe-message> urlscan<Enter>" "call urlscan to - extract URLs out of a message" - - macro attach,compose \\cb "<pipe-entry> urlscan<Enter>" "call urlscan to - extract URLs out of a message" - -Once this is done, Control-b while reading mail in mutt will automatically -invoke urlscan on the message. - -To choose a particular browser, set the environment variable BROWSER. If BROWSER -is not set, xdg-open will control which browser is used, if it's available.: - - export BROWSER=/usr/bin/epiphany - - -Command Line usage ------------------- - -:: - - urlscan [-g, --genconf] [-n, --no-browser] [-c, --compact] [-d, --dedupe] [--headers] [-r, --run <expression>] [-f, --run-safe <expression>] [-R, --reverse] [-s, --single] [-p, --pipe] [-w, --width] [-H, --nohelp] [-E, --regex <expression>] <file> - -Urlscan can extract URLs and email addresses from emails or any text file. -Calling with no flags will start the curses browser. Calling with '-n' will just -output a list of URLs/email addressess to stdout. The '-c' flag removes the -context from around the URLs in the curses browser, and the '-d' flag removes -duplicate URLs. 'R' reverses the displayed order of URLs and context. Files can -also be piped to urlscan using normal shell pipe mechanisms: `cat <something> | -urlscan` or `urlscan < <something>` - -Instead of opening a web browser, the selected URL can be passed as the argument -to a command using `--run-safe "<command> {}"` or `--run "<command> {}"`. Note -the use of `{}` in the command string to denote the selected URL. Alternatively, -the URL can be piped to the command using `--run-safe <command> --pipe` (or -`--run`). Using --run-safe with --pipe is preferred if the command supports it, -as it is marginally more secure and tolerant of special characters in the URL. - -Theming -------- - -Run `urlscan -g` to generate ~/.config/urlscan/config.json with the default -color and black & white palettes. This can be edited or added to, as desired. -The first palette in the list will be the default. Configure the palettes -according to the `Urwid display attributes`_. - -Display width can be set with `--width`. - -Keybindings ------------ - -Run `urlscan -g` to generate ~/.config/urlscan/config.json. All of the keys will -be listed. You can either leave in place or delete any that will not be altered. - -To unset a binding, set it equal to "". For example: `"P": ""` - -The follow actions are supported: - -- `all_escape` -- toggle unescape all URLs (default: `u`) -- `all_shorten` -- toggle shorten all URLs (default: `S`) -- `bottom` -- move cursor to last item (default: `G`) -- `clear_screen` -- redraw screen (default: `Ctrl-l`) -- `clipboard` -- copy highlighted URL to clipboard using xsel/xclip (default: `C`) -- `clipboard_pri` -- copy highlighted URL to primary selection using xsel/xclip (default: `P`) -- `context` -- show/hide context (default: `c`) -- `down` -- cursor down (default: `j`) -- `help_menu` -- show/hide help menu (default: `F1`) -- `link_handler` -- cycle link handling (webbrowser, xdg-open, --run-safe or --run) (default: `l`) -- `open_url` -- open selected URL (default: `space` or `enter`) -- `palette` -- cycle through palettes (default: `p`) -- `quit` -- quit (default: `q` or `Q`) -- `reverse` -- reverse display order (default: `R`) -- `shorten` -- toggle shorten highlighted URL (default: `s`) -- `top` -- move to first list item (default: `g`) -- `up` -- cursor up (default: `k`) - -Update TLD list (for developers, not users) -------------------------------------------- - -`wget https://data.iana.org/TLD/tlds-alpha-by-domain.txt` - -Known bugs and limitations --------------------------- - -- Running urlscan sometimes "messes up" the terminal background. This seems to - be an urwid bug, but I haven't tracked down just what's going on. - -- Extraction of context from HTML messages leaves something to be desired. - Probably the ideal solution would be to extract context on a word basis rather - than on a paragraph basis. - -- The HTML message handling is a bit kludgy in general. - -- multipart/alternative sections are handled by descending into all the - sub-parts, rather than just picking one, which may lead to URLs and context - appearing twice. (Bypass this by selecting the '--dedupe' option) - -.. _Urwid display attributes: http://urwid.org/manual/displayattributes.html#display-attributes diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/urlscan-0.9.6/bin/urlscan new/urlscan-0.9.7/bin/urlscan --- old/urlscan-0.9.6/bin/urlscan 2021-03-23 06:00:17.000000000 +0100 +++ new/urlscan-0.9.7/bin/urlscan 1970-01-01 01:00:00.000000000 +0100 @@ -1,187 +0,0 @@ -#!/usr/bin/env python3 -""" A simple urlview replacement that handles things like quoted-printable -properly. - -""" -# -# Copyright (C) 2006-2007 Daniel Burrows -# Copyright (C) 2021 Scott Hansen -# -# This program is free software; you can redistribute it and/or -# modify it under the terms of the GNU General Public License -# as published by the Free Software Foundation; either version 2 -# of the License, or (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with this program; if not, write to the Free Software -# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA - -import argparse -import io -import os -import sys -from email import policy -from email.parser import BytesParser -from urlscan import urlchoose, urlscan - - -def parse_arguments(): - """Parse command line options. - - Returns: args - - """ - arg_parse = argparse.ArgumentParser(description="Parse and display URLs") - arg_parse.add_argument('--genconf', '-g', - action='store_true', default=False, - help="Generate config file and exit.") - arg_parse.add_argument('--compact', '-c', - action='store_true', default=False, - help="Don't display the context of each URL.") - arg_parse.add_argument('--reverse', '-R', dest="reverse", - action='store_true', default=False, - help="Reverse order of displayed URLS/context") - arg_parse.add_argument('--no-browser', '-n', dest="nobrowser", - action='store_true', default=False, - help="Pipe URLs to stdout") - arg_parse.add_argument('--dedupe', '-d', dest="dedupe", - action='store_true', default=False, - help="Remove duplicate URLs from list") - arg_parse.add_argument('--regex', '-E', - help="Alternate custom regex to be used for all " - "kinds of matching. " - "For example: --regex 'https?://.+\.\w+'") - arg_parse.add_argument('--run', '-r', - help="Alternate command to run on selected URL " - "instead of opening URL in browser. Use {} to " - "represent the URL value in the expression. " - "For example: --run 'echo {} | xclip -i'") - arg_parse.add_argument('--run-safe', '-f', dest="runsafe", - help="Alternate command to run on selected URL " - "instead of opening URL in browser. Use {} to " - "represent the URL value in the expression. Safest " - "run option but uses `shell=False` which does not " - "allow use of shell features like | or >. Can use " - "with --pipe.") - arg_parse.add_argument('--pipe', '-p', dest='pipe', - action='store_true', default=False, - help="Pipe URL into the command specified by --run or --run-safe") - arg_parse.add_argument('--nohelp', '-H', dest='nohelp', - action='store_true', default=False, - help='Hide help menu by default') - arg_parse.add_argument('--single', '-s', dest='single', - action='store_true', default=False, - help='Quit urlscan after opening/copying a single link.') - arg_parse.add_argument('--width', '-w', dest='width', - type=int, default=0, - help='Set width to display') - arg_parse.add_argument('--headers', dest='headers', - action='store_true', default=False, - help='Scan certain message headers for URLs.') - arg_parse.add_argument('message', nargs='?', default=sys.stdin, - help="Filename of the message to parse") - return arg_parse.parse_args() - - -def close_stdin(): - """This section closes out sys.stdin if necessary so as not to block curses - keyboard inputs - - """ - if not os.isatty(0): - fdesc = os.open('/dev/tty', os.O_RDONLY) - if fdesc < 0: - sys.stderr.write('Unable to open an input tty.\n') - sys.exit(-1) - else: - os.dup2(fdesc, 0) - os.close(fdesc) - - -def process_input(fname): - """Return the parsed text of stdin or the message. Accounts for possible - file encoding differences. - - Args: fname - filename or sys.stdin - Returns: mesg - EmailMessage object - - """ - if fname is sys.stdin: - try: - stdin_file = fname.buffer.read() - except AttributeError: - stdin_file = fname.read() - else: - stdin_file = None - if stdin_file is not None: - fobj = io.BytesIO(stdin_file) - else: - fobj = io.open(fname, mode='rb') - f_keep = fobj - mesg = BytesParser(policy=policy.default.clone(utf8=True)).parse(fobj) - if 'From' not in mesg.keys() and 'Date' not in mesg.keys(): - # If it's not an email message, don't let the email parser - # delete the first line. If it is, let the parser do its job so - # we don't get mailto: links for all the To and From addresses - fobj = _fix_first_line(f_keep) - mesg = BytesParser(policy=policy.default.clone(utf8=True)).parse(fobj) - try: - fobj.close() - except NameError: - pass - close_stdin() - return mesg - - -def _fix_first_line(fline): - """If the first line starts with http* or [ or other non-text characters, - the URLs on that line will not be parsed by email.Parser. Add a blank line - at the top of the file to ensure everything is read in a non-email file. - - """ - fline.seek(0) - new = io.BytesIO() - new.write(b"\n" + fline.read()) - fline.close() - new.seek(0) - return new - - -def main(): - """Entrypoint function for urlscan - - """ - args = parse_arguments() - if args.genconf is True: - urlchoose.URLChooser([], genconf=True) - return - msg = process_input(args.message) - if args.nobrowser is False: - tui = urlchoose.URLChooser(urlscan.msgurls(msg, regex=args.regex, headers=args.headers), - compact=args.compact, - reverse=args.reverse, - nohelp=args.nohelp, - dedupe=args.dedupe, - run=args.run, - runsafe=args.runsafe, - single=args.single, - width=args.width, - pipe=args.pipe) - tui.main() - else: - out = urlchoose.URLChooser(urlscan.msgurls(msg, regex=args.regex, headers=args.headers), - dedupe=args.dedupe, - reverse=args.reverse, - shorten=False) - if args.reverse is True: - out.urls.reverse() - print("\n".join(out.urls)) - - -if __name__ == "__main__": - main() diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/urlscan-0.9.6/setup.py new/urlscan-0.9.7/setup.py --- old/urlscan-0.9.6/setup.py 2021-03-23 06:00:17.000000000 +0100 +++ new/urlscan-0.9.7/setup.py 2021-10-06 20:10:59.000000000 +0200 @@ -1,18 +1,30 @@ #!/usr/bin/env python3 +"""urlscan setup.py""" from setuptools import setup + +def long_description(): + """Generate long description from README""" + with open("README.md") as readme: + return readme.read() + + setup(name="urlscan", - version="0.9.6", + version="0.9.7", description="View/select the URLs in an email message or file", + long_description=long_description(), + long_description_content_type="text/markdown", author="Scott Hansen", author_email="[email protected]", url="https://github.com/firecat53/urlscan";, download_url="https://github.com/firecat53/urlscan/archive/0.9.6.zip";, packages=['urlscan'], - scripts=['bin/urlscan'], + entry_points={ + 'console_scripts': ['urlscan=urlscan.__main__:main'] + }, package_data={'urlscan': ['assets/*']}, - data_files=[('share/doc/urlscan', ['README.rst', 'COPYING']), + data_files=[('share/doc/urlscan', ['README.md', 'COPYING']), ('share/man/man1', ['urlscan.1'])], license="GPLv2", install_requires=["urwid>=1.2.1"], @@ -28,5 +40,5 @@ 'Programming Language :: Python :: 3.8', 'Programming Language :: Python :: 3.9', 'Topic :: Utilities'], - keywords=("urlscan urlview email mutt tmux"), + keywords="urlscan, urlview, email, mutt, tmux" ) diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/urlscan-0.9.6/urlscan/__main__.py new/urlscan-0.9.7/urlscan/__main__.py --- old/urlscan-0.9.6/urlscan/__main__.py 1970-01-01 01:00:00.000000000 +0100 +++ new/urlscan-0.9.7/urlscan/__main__.py 2021-10-06 20:10:59.000000000 +0200 @@ -0,0 +1,197 @@ +#!/usr/bin/env python3 +""" A simple urlview replacement that handles things like quoted-printable +properly. + +""" +# +# Copyright (C) 2006-2007 Daniel Burrows +# Copyright (C) 2021 Scott Hansen +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU General Public License +# as published by the Free Software Foundation; either version 2 +# of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA + +import argparse +import io +import os +import sys +from email import policy +from email.parser import BytesParser +from urlscan import urlchoose, urlscan + + +def parse_arguments(): + """Parse command line options. + + Returns: args + + """ + arg_parse = argparse.ArgumentParser(description="Parse and display URLs") + arg_parse.add_argument('--genconf', '-g', + action='store_true', default=False, + help="Generate config file and exit.") + arg_parse.add_argument('--compact', '-c', + action='store_true', default=False, + help="Don't display the context of each URL.") + arg_parse.add_argument('--reverse', '-R', dest="reverse", + action='store_true', default=False, + help="Reverse order of displayed URLS/context") + arg_parse.add_argument('--no-browser', '-n', dest="nobrowser", + action='store_true', default=False, + help="Pipe URLs to stdout") + arg_parse.add_argument('--dedupe', '-d', dest="dedupe", + action='store_true', default=False, + help="Remove duplicate URLs from list") + arg_parse.add_argument('--regex', '-E', + help="Alternate custom regex to be used for all " + "kinds of matching. " + "For example: --regex 'https?://.+\.\w+'") + arg_parse.add_argument('--run', '-r', + help="Alternate command to run on selected URL " + "instead of opening URL in browser. Use {} to " + "represent the URL value in the expression. " + "For example: --run 'echo {} | xclip -i'") + arg_parse.add_argument('--run-safe', '-f', dest="runsafe", + help="Alternate command to run on selected URL " + "instead of opening URL in browser. Use {} to " + "represent the URL value in the expression. Safest " + "run option but uses `shell=False` which does not " + "allow use of shell features like | or >. Can use " + "with --pipe.") + arg_parse.add_argument('--pipe', '-p', dest='pipe', + action='store_true', default=False, + help="Pipe URL into the command specified by --run or --run-safe") + arg_parse.add_argument('--nohelp', '-H', dest='nohelp', + action='store_true', default=False, + help='Hide help menu by default') + arg_parse.add_argument('--single', '-s', dest='single', + action='store_true', default=False, + help='Quit urlscan after opening/copying a single link.') + arg_parse.add_argument('--width', '-w', dest='width', + type=int, default=0, + help='Set width to display') + arg_parse.add_argument('--whitespace-off', '-W', dest='whitespaceoff', + action='store_true', default=False, + help="Don't display empty lines and ellipses.") + arg_parse.add_argument('--headers', dest='headers', + action='store_true', default=False, + help='Scan certain message headers for URLs.') + arg_parse.add_argument('message', nargs='?', default=sys.stdin, + help="Filename of the message to parse") + return arg_parse.parse_args() + + +def close_stdin(): + """This section closes out sys.stdin if necessary so as not to block curses + keyboard inputs + + """ + if not os.isatty(0): + try: + fdesc = os.open('/dev/tty', os.O_RDONLY) + except OSError: + # This is most likely a non-interactive session, try to open + # `stdin` directly + fdesc = os.open('/dev/stdin', os.O_RDONLY) + + if fdesc < 0: + sys.stderr.write('Unable to open an input tty.\n') + sys.exit(-1) + else: + os.dup2(fdesc, 0) + os.close(fdesc) + + +def process_input(fname): + """Return the parsed text of stdin or the message. Accounts for possible + file encoding differences. + + Args: fname - filename or sys.stdin + Returns: mesg - EmailMessage object + + """ + if fname is sys.stdin: + try: + stdin_file = fname.buffer.read() + except AttributeError: + stdin_file = fname.read() + else: + stdin_file = None + if stdin_file is not None: + fobj = io.BytesIO(stdin_file) + else: + fobj = io.open(fname, mode='rb') + f_keep = fobj + mesg = BytesParser(policy=policy.default.clone(utf8=True)).parse(fobj) + if 'From' not in mesg.keys() and 'Date' not in mesg.keys(): + # If it's not an email message, don't let the email parser + # delete the first line. If it is, let the parser do its job so + # we don't get mailto: links for all the To and From addresses + fobj = _fix_first_line(f_keep) + mesg = BytesParser(policy=policy.default.clone(utf8=True)).parse(fobj) + try: + fobj.close() + except NameError: + pass + close_stdin() + return mesg + + +def _fix_first_line(fline): + """If the first line starts with http* or [ or other non-text characters, + the URLs on that line will not be parsed by email.Parser. Add a blank line + at the top of the file to ensure everything is read in a non-email file. + + """ + fline.seek(0) + new = io.BytesIO() + new.write(b"\n" + fline.read()) + fline.close() + new.seek(0) + return new + + +def main(): + """Entrypoint function for urlscan + + """ + args = parse_arguments() + if args.genconf is True: + urlchoose.URLChooser([], genconf=True) + return + msg = process_input(args.message) + if args.nobrowser is False: + tui = urlchoose.URLChooser(urlscan.msgurls(msg, regex=args.regex, headers=args.headers), + compact=args.compact, + reverse=args.reverse, + nohelp=args.nohelp, + dedupe=args.dedupe, + run=args.run, + runsafe=args.runsafe, + single=args.single, + width=args.width, + whitespaceoff=args.whitespaceoff, + pipe=args.pipe) + tui.main() + else: + out = urlchoose.URLChooser(urlscan.msgurls(msg, regex=args.regex, headers=args.headers), + dedupe=args.dedupe, + reverse=args.reverse, + shorten=False) + if args.reverse is True: + out.urls.reverse() + print("\n".join(out.urls)) + + +if __name__ == "__main__": + main() diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/urlscan-0.9.6/urlscan/urlchoose.py new/urlscan-0.9.7/urlscan/urlchoose.py --- old/urlscan-0.9.6/urlscan/urlchoose.py 2021-03-23 06:00:17.000000000 +0100 +++ new/urlscan-0.9.7/urlscan/urlchoose.py 2021-10-06 20:10:59.000000000 +0200 @@ -93,7 +93,7 @@ def __init__(self, extractedurls, compact=False, reverse=False, nohelp=False, dedupe=False, shorten=True, run="", runsafe="", single=False, pipe=False, - genconf=False, width=0): + genconf=False, width=0, whitespaceoff=False): self.conf = expanduser("~/.config/urlscan/config.json") self.keys = {'/': self._search_key, '0': self._digits, @@ -145,7 +145,17 @@ ('urlref:number', 'white', 'black', 'standout'), ('urlref:url', 'white', 'black', 'standout'), ('url:sel', 'black', 'light gray', 'bold')] - self.palettes.update([("default", default), ("bw", blw)]) + # Boruch's colorized palette + colorized =[('header','brown','black','standout'), + ('footer','white','dark red','standout'), + ('search','white','dark green','standout'), + ('msgtext','light cyan','black'), + ('msgtext:ellipses','light gray','black'), + ('urlref:number:braces','light gray','black'), + ('urlref:number','yellow','black','standout'), + ('urlref:url','dark green','black','standout'), + ('url:sel','white','black','')] + self.palettes.update([("default", default), ("bw", blw), ("colorized", colorized)]) if genconf is True: self._config_create() try: @@ -188,6 +198,7 @@ self.enter = False self.term_width, _ = urwid.raw_display.Screen().get_cols_rows() self.width = min(self.term_width, width or self.term_width) + self.whitespaceoff = whitespaceoff self.activate_keys = [i for i, j in urwid.Button._command_map._command.items() if j == 'activate'] self.items, self.urls = self.process_urls(extractedurls, @@ -341,27 +352,31 @@ '_digits']) help_men = "KEYBINDINGS\n" + help_men + "\n<0-9> - Jump to item" docs = ("OPTIONS\n" - "all_escape -- toggle unescape all URLs\n" - "all_shorten -- toggle shorten all URLs\n" - "bottom -- move cursor to last item\n" - "clear_screen -- redraw screen\n" - "clipboard -- copy highlighted URL to clipboard using xsel/xclip\n" - "clipboard_pri -- copy highlighted URL to primary selection using xsel/xclip\n" + "all_escape -- toggle unescape all URLs\n" + "all_shorten -- toggle shorten all URLs\n" + "bottom -- move cursor to last item\n" + "clear_screen -- redraw screen\n" + "clipboard -- copy highlighted URL to clipboard\n" + " using xsel/xclip\n" + "clipboard_pri -- copy highlighted URL to primary\n" + " selection using xsel/xclip\n" "config_create -- create ~/.config/urlscan/config.json\n" - "context -- show/hide context\n" - "down -- cursor down\n" - "help_menu -- show/hide help menu\n" - "link_handler -- cycle through xdg-open, webbrowser and user-defined function\n" - "open_url -- open selected URL\n" - "palette -- cycle through palettes\n" - "quit -- quit\n" - "reverse -- reverse order URLs/context\n" - "shorten -- toggle shorten highlighted URL\n" - "single -- quit urlscan after opening a single link\n" - "top -- move to first list item\n" - "up -- cursor up\n") + "context -- show/hide context\n" + "down -- cursor down\n" + "help_menu -- show/hide help menu\n" + "link_handler -- cycle through xdg-open, webbrowser \n" + " and user-defined function\n" + "open_url -- open selected URL\n" + "palette -- cycle through palettes\n" + "quit -- quit\n" + "reverse -- reverse order URLs/context\n" + "shorten -- toggle shorten highlighted URL\n" + "single -- quit urlscan after opening a\n" + " single link\n" + "top -- move to first list item\n" + "up -- cursor up\n") self.top.base_widget.body = \ - urwid.ListBox(urwid.SimpleListWalker([urwid.Columns([(30, urwid.Text(help_men)), + urwid.ListBox(urwid.SimpleListWalker([urwid.Columns([(24, urwid.Text(help_men)), urwid.Text(docs)])])) else: self.top.base_widget.body = urwid.ListBox(self.items) @@ -663,7 +678,7 @@ self.search = False self.enter = False elif self.link_open_modes[0] == "Web Browser": - webbrowser.open(url.replace('&', '\&')) + webbrowser.open(url) elif self.link_open_modes[0] == "Xdg-Open": run = 'xdg-open "{}"'.format(url) process = Popen(shlex.split(run), stdout=PIPE, stdin=PIPE) @@ -707,7 +722,7 @@ continue groupurls = [] markup = [] - if not usedfirst: + if not usedfirst and not self.whitespaceoff: markup.append(('msgtext:ellipses', '...\n')) for chunks in group: i = 0 @@ -736,8 +751,9 @@ ('urlref:number:braces', ' ['), ('urlref:number', repr(url_idx)), ('urlref:number:braces', ']')] - markup += '\n' - if not usedlast: + if not self.whitespaceoff: + markup += '\n' + if not usedlast and not self.whitespaceoff: markup += [('msgtext:ellipses', '...\n\n')] items.append(urwid.Text(markup)) diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/urlscan-0.9.6/urlscan.1 new/urlscan-0.9.7/urlscan.1 --- old/urlscan-0.9.6/urlscan.1 2021-03-23 06:00:17.000000000 +0100 +++ new/urlscan-0.9.7/urlscan.1 2021-10-06 20:10:59.000000000 +0200 @@ -68,22 +68,21 @@ .SH OPTIONS .TP -.B \-g, \-\-genconf -Generate ~/.config/urlscan/config.json with default options. -.TP .B \-c, \-\-compact Display a simple list of the extracted URLs, instead of showing the context of each URL. Also toggle with `c` from within the viewer. .TP -.B \-H, \-\-nohelp -Start with header menu hidden. -.TP .B \-d, \-\-dedupe Remove duplicated URLs from the list of URLs. .TP -.B \-n, \-\-no-browser -Disables the selection interface and print the links to standard output. -Useful for scripting (implies \fB\-\-compact\fR). +.B \-E, \-\-regex \<expression\> +Use \<expression\> in place of the default set of regular expressions, +to be used for any kind of matching. This is useful for example when +selectively avoiding 'mailto:' links or any other pattern that urlscan +could interpret as urls (such as '<filename>.<extension>'). Usage +example: + + $ urlscan --regex 'https?://.+\.\w+' file.txt .TP .B \-f, \-\-run\-safe \<expression\> Execute \<expression\> in place of opening URL with a browser. Use {} in @@ -91,12 +90,19 @@ $ urlscan --run-safe 'tmux set buffer {}' .TP -.B \-r, \-\-run \<expression\> -Execute \<expression\> in place of opening URL with a browser. Use {} in -\<expression\> to substitute in the URL. Shell features such as \| and \> can be -used, but it is less secure. Examples: +.B \-g, \-\-genconf +Generate ~/.config/urlscan/config.json with default options. +.TP +.B \-H, \-\-nohelp +Start with header menu hidden. +.TP +.B \-\-headers +Scan email headers for URLs. - $ urlscan --run 'echo {} | xclip -i' file.txt +.TP +.B \-n, \-\-no-browser +Disables the selection interface and print the links to standard output. +Useful for scripting (implies \fB\-\-compact\fR). .TP .B \-p, \-\-pipe Pipe the selected URL to the command specified by `--run-safe` or `--run`. This @@ -108,6 +114,13 @@ .B \-R, \-\-reverse Reverse displayed order of URLs. .TP +.B \-r, \-\-run \<expression\> +Execute \<expression\> in place of opening URL with a browser. Use {} in +\<expression\> to substitute in the URL. Shell features such as \| and \> can be +used, but it is less secure. Examples: + + $ urlscan --run 'echo {} | xclip -i' file.txt +.TP .B \-s, \-\-single Exit urlscan after opening or copying a single browser link. @@ -116,17 +129,8 @@ .B \-w, \-\-width Set display width. .TP -.B \-E, \-\-regex \<expression\> -Use \<expression\> in place of the default set of regular expressions, -to be used for any kind of matching. This is useful for example when -selectively avoiding 'mailto:' links or any other pattern that urlscan -could interpret as urls (such as '<filename>.<extension>'). Usage -example: - - $ urlscan --regex 'https?://.+\.\w+' file.txt -.TP -.B \-\-headers -Scan email headers for URLs. +.B \-W, \-\-whitespace-off +Suppress output of blank lines and ellipses lines. .SH MUTT INTEGRATION @@ -196,7 +200,7 @@ $HOME/.config/urlscan/config.json -Only required if additional or modified palettes are desired. +Only required if additional or modified palettes or keybindings are desired. .SH SEE ALSO \fI/usr/share/doc/urlscan/README\fR,
