Discussion:
[sbase] Command list
(too old to reply)
sin
2013-10-17 17:38:54 UTC
Permalink
Hi all,

I've been thinking, besides the commands that are already listed in TODO
and all the unimplemented options for existing commands, what other commands
do we want to include in sbase?

The current TODO list includes:

diff, expr, printf, tr, unexpand

What about:

od, logname, uuencode, uudecode, mktemp, passwd, su, sed, patch

Let me know what you guys think.

I think all of these can be implemented portably, otherwise, they are a
candidate for ubase.

Thanks,
sin
Roberto E. Vargas Caballero
2013-10-17 18:15:16 UTC
Permalink
Post by sin
od, logname, uuencode, uudecode, mktemp, passwd, su, sed, patch
ed and awk are also missed (yeah, I know they are bigger and
more difficult programs, but they are basic for a system).
--
Roberto E. Vargas Caballero
_______________________________________________________________________
'Write programs that do one thing and do it well. Write programs to work
together. Write programs to handle text streams, because that is a
universal interface'

(Doug McIlroy)

In Other Words - Don't design like polkit or systemd
_______________________________________________________________________
Alex Pilon
2013-10-17 19:36:12 UTC
Permalink
Post by sin
I've been thinking, besides the commands that are already listed in TODO
and all the unimplemented options for existing commands, what other commands
do we want to include in sbase?
[
]
od, logname, uuencode, uudecode, mktemp, passwd, su, sed, patch
Don't od's and hexdump's functionality overlap?

Do we still really need uuencode and uudecode?

Regards,

Alex Pilon
Thorsten Glaser
2013-10-17 19:46:35 UTC
Permalink
Post by Alex Pilon
Don't od's and hexdump's functionality overlap?
Yes, they (and hd) are normally one binary.
Post by Alex Pilon
Do we still really need uuencode and uudecode?
Yes!

I often use them, in combination with GNU screen, to
copy/paste files(!) from one tab to the other, when
using other methods (such as rsync or scp/sftp) is
unpractical or impossible (think serial console).

bye,
//mirabilos
--=20
<Natureshadow> Boah, ich steck hier soviel Anstrengung
ins Furzen, da=C3=9F ich dabei Schwindelanf=C3=A4lle krieg
Alex Pilon
2013-10-17 23:00:16 UTC
Permalink
Post by Thorsten Glaser
Post by Alex Pilon
Do we still really need uuencode and uudecode?
Yes!
I often use them, in combination with GNU screen, to
copy/paste files(!) from one tab to the other, when
using other methods (such as rsync or scp/sftp) is
unpractical or impossible (think serial console).
I know the trick. I used base64 for that, not uuencode. Don't they both have the
same overhead, other than uuencode's header? That header's redundant anyway, at
least for this hack, since you can just use tar or cpio.

Regards,

Alex Pilon
Thorsten Glaser
2013-10-17 23:22:48 UTC
Permalink
Post by Alex Pilon
I know the trick. I used base64 for that, not uuencode. Don't they
both have the same overhead, other than uuencode's header?
Yes but uuencode is more portable=E2=80=A6 e.g. the Linux =E2=80=9Cbase64=
=E2=80=9D tool is
rarely installed anywhere and is extremely picky about the formatting
of its input; OpenBSD has b64decode but it demands header lines similar
to uuencode (MirBSD has b64decode -r which is not picky either =E2=80=93 bu=
t
that=E2=80=99s only in MirBSD of course).

Nowadays, uuencode (=E2=80=9Csharutils=E2=80=9D) is not installed everywher=
e any more
by default, but it=E2=80=99s still the most widespread portable one and usu=
ally
easy to get if it=E2=80=99s not yet there.

Meh, maybe I should write a shell uu{en,de}code implementation and
put it into the standard .mkshrc I ship=E2=80=A6

bye,
//mirabilos
--=20
This space for rent.
Truls Becken
2013-10-18 09:05:07 UTC
Permalink
A few more suggestions:

bc dc dd file find fmt install killall less make sh time which xargs
(un)zip/gzip/bzip2/lzma/xz

Some of these are more complex than others of course.

-Truls
Chris Down
2013-10-18 09:11:30 UTC
Permalink
sh
There have already been a few discussions about the sbase shell, I believe the
last opinion given was that we shouldn't include one.
which
This should be left to the shell usually, since almost everyone uses a hash
table for lookups nowadays, and we have no idea about what's in it or how it
caches.
sin
2013-10-18 10:29:11 UTC
Permalink
Post by Truls Becken
bc dc dd file find fmt install killall less make sh time which xargs
(un)zip/gzip/bzip2/lzma/xz
dd: I suspect this might not be portable if we want an optimized version.
find: Useless, just do `du -a | grep blabla'
killall: Not portable, candidate for ubase, not sure we even need it.
make: Do we really need this in sbase?
sh: We agreed to do without a shell.
xargs: Why?
(un)zip: Why?
gzip/bzip2/lzma/xz: Maybe as separate programs and not part of sbase?

Thanks,
sin
Roberto E. Vargas Caballero
2013-10-18 11:40:29 UTC
Permalink
Post by sin
dd: I suspect this might not be portable if we want an optimized version.
find: Useless, just do `du -a | grep blabla'
and a 'find -type d -name xxxx -prune -o -type f -perm g=x -exec chmod g-x {} \; '?
find is a mini language for run over directories, and it is a must.
Post by sin
xargs: Why?
if you have a directory with a very, very, very, very large number of files and
then you try a 'rm *' you will see why you need xargs. I use it a lot in combination
with find. For example in the previous example a better way (faster):

find -type d -name xxxx -prune -o -type f -perm g=x | xargs chmod g-x

Regards,
--
Roberto E. Vargas Caballero
_______________________________________________________________________
'Write programs that do one thing and do it well. Write programs to work
together. Write programs to handle text streams, because that is a
universal interface'

(Doug McIlroy)

In Other Words - Don't design like polkit or systemd
_______________________________________________________________________
sin
2013-10-18 12:01:05 UTC
Permalink
Post by Roberto E. Vargas Caballero
Post by sin
dd: I suspect this might not be portable if we want an optimized version.
find: Useless, just do `du -a | grep blabla'
and a 'find -type d -name xxxx -prune -o -type f -perm g=x -exec chmod g-x {} \; '?
find is a mini language for run over directories, and it is a must.
Not UNIX.
Post by Roberto E. Vargas Caballero
Post by sin
xargs: Why?
if you have a directory with a very, very, very, very large number of files and
then you try a 'rm *' you will see why you need xargs. I use it a lot in combination
I realize when xargs is useful, I just hate it.
Roberto E. Vargas Caballero
2013-10-18 12:04:06 UTC
Permalink
Post by sin
Post by Roberto E. Vargas Caballero
Post by sin
dd: I suspect this might not be portable if we want an optimized version.
find: Useless, just do `du -a | grep blabla'
and a 'find -type d -name xxxx -prune -o -type f -perm g=x -exec chmod g-x {} \; '?
find is a mini language for run over directories, and it is a must.
Not UNIX.
Not UNIX?, what do you mean?. Sorry, I don't understand what you want say here.
--
Roberto E. Vargas Caballero
_______________________________________________________________________
'Write programs that do one thing and do it well. Write programs to work
together. Write programs to handle text streams, because that is a
universal interface'

(Doug McIlroy)

In Other Words - Don't design like polkit or systemd
_______________________________________________________________________
sin
2013-10-18 12:07:41 UTC
Permalink
Post by Roberto E. Vargas Caballero
Post by sin
Post by Roberto E. Vargas Caballero
Post by sin
dd: I suspect this might not be portable if we want an optimized version.
find: Useless, just do `du -a | grep blabla'
and a 'find -type d -name xxxx -prune -o -type f -perm g=x -exec chmod g-x {} \; '?
find is a mini language for run over directories, and it is a must.
Not UNIX.
Not UNIX?, what do you mean?. Sorry, I don't understand what you want say here.
I do not consider find(1) to be elegant in any way. I agree many ppl use it,
including myself, I just wish we could do without this madness.
Raphaël Proust
2013-10-18 12:18:20 UTC
Permalink
Post by sin
I do not consider find(1) to be elegant in any way. I agree many ppl use it,
including myself, I just wish we could do without this madness.
Maybe we need a way to filter lists of file names:
ls | ff -type d | ff -name xxxx


Seems more compositional… Unixier one might say.
--
______________
Raphaël Proust
Roberto E. Vargas Caballero
2013-10-18 12:18:37 UTC
Permalink
Post by sin
I do not consider find(1) to be elegant in any way. I agree many ppl use it,
including myself, I just wish we could do without this madness.
Well, we can not agree in all the points ;). I see find like the best example of
the eleganty of Unix, do one thing and do it well. Other programs should not
run over directories, and the best example of this problem is tar, which
needs a lot of different options in order to select the correct files, instead
of doing it like cpio, which reads the input list of files from stdin, so
you can use a highly specialized tool for it, that it is of course find.

Regards,
--
Roberto E. Vargas Caballero
_______________________________________________________________________
'Write programs that do one thing and do it well. Write programs to work
together. Write programs to handle text streams, because that is a
universal interface'

(Doug McIlroy)

In Other Words - Don't design like polkit or systemd
_______________________________________________________________________
sin
2013-10-18 12:34:09 UTC
Permalink
Post by Roberto E. Vargas Caballero
Post by sin
I do not consider find(1) to be elegant in any way. I agree many ppl use it,
including myself, I just wish we could do without this madness.
Well, we can not agree in all the points ;). I see find like the best example of
the eleganty of Unix, do one thing and do it well. Other programs should not
run over directories, and the best example of this problem is tar, which
needs a lot of different options in order to select the correct files, instead
of doing it like cpio, which reads the input list of files from stdin, so
you can use a highly specialized tool for it, that it is of course find.
As this seems to upset the sbase audience, a small sensible subset of the find(1)
functionality can be implemented for sbase I suppose.
Markus Wichmann
2013-10-18 20:32:10 UTC
Permalink
Post by sin
I do not consider find(1) to be elegant in any way. I agree many ppl use it,
including myself, I just wish we could do without this madness.
I can do without it! I just do

setopt extendedglob
chmod g-x **/*(#q.E^AI)~xxxx/**/*

(I hope I understood that find command correctly)

The only possible prolem would arise from the argumen list being too
long, and then I can just

zmodload zsh/files

which implements chmod inside of zsh. Or I could

print -l **/*(#q.E^AI)~xxxx/**/* | xargs chmod g-x

But of course that requires the zsh. So there's at least something all
that bloated code buys.

Ciao,
Markus
Alexander S.
2013-10-19 01:00:44 UTC
Permalink
Contributing into the discussion about find.
Most of its selectors are implemented by stest(1); commonly useful
ones that I can think of are -user and -group. However:
1) ostracize me, but I like find "one dash, long option" style way
more than stest "a thousand of letters a-la test". I understand the
test(1) legacy here, I just say that it wouldn't be very pleasant to
consult with manual each time I'm trying to write a command.
2) there is no way to express intersection, union and negation of a
test. They are mostly irrelevant for test(1), but here we ought to
have at least disjunction, because making temporary lists and merging
them with sort -u or comm(1) is unpleasant to say the least. (Also,
slow). Theoretically speaking, every logical expression can be
represented in conjunctive normal form, so providing support for
negation and union and using multiple commands & pipes as intersection
("| stest test1 | stest test2") should be enough, but in some cases,
unpleasant too.
3) we need a program that can print a directory tree well, with
-mindepth, -maxdepth, and sorted/unsorted options.
4) we need some standard way to separate file names in a stream.
Basically, ^@ is our only refuge, because path can contain any other
character. But maybe newline is good enough?
5) we need a program that can take input and convert into a
commandline. So, xargs (but of course without space-separation
quotation nonsense. Not once nor twice had I written the glorious "tr
'\n' '\0' | xargs -0" to turn it off). I suppose most of POSIX options
are worth keeping.

So, the former "find" command would be like
ptree -maxdepth 2 | ftest "( perm 111 || newer ./reference ) && user
root" | xargs rm
I feel that going the awk way (e. g. inventing the mini-language and
passing expression as one string) is better than find way, with -a and
-o and escaped parens.
--
Best regards, Alexander.
Alex Pilon
2013-10-19 15:47:47 UTC
Permalink
Post by Alexander S.
4) we need some standard way to separate file names in a stream.
character. But maybe newline is good enough?
Why not just play it safe? Would it really add any noticeable
complexity, even if just an option?
Post by Alexander S.
Not once nor twice had I written the glorious "tr '\n' '\0' | xargs -0"
to turn it off).
Which does nothing for safety. `xargs -d '\n'` would have been just as
good.
Post by Alexander S.
ptree -maxdepth 2 | ftest "( perm 111 || newer ./reference ) && user root" | xargs rm
The Solaris and SunOS users *might* not like you for that choice of name
(“ptree”), but who cares? It's inconsequential.

Regards,

Alex Pilon
Chris Down
2013-10-19 16:49:43 UTC
Permalink
Post by Alex Pilon
Post by Alexander S.
4) we need some standard way to separate file names in a stream.
character. But maybe newline is good enough?
Why not just play it safe? Would it really add any noticeable
complexity, even if just an option?
In my opinion, having no way to write secure programs using these tools
due to this would undermine all the great work being done in sbase.
Truls Becken
2013-10-18 12:03:27 UTC
Permalink
Post by sin
find: Useless, just do `du -a | grep blabla'
I'm not interested in disk usage, but finding files based on certain
properties, such as update time, ownership, permissions, etc.
Post by sin
make: Do we really need this in sbase?
Make is a basic tool, useful in lots of situations.
It would be nice not to rely on GNU for this.
Post by sin
sh: We agreed to do without a shell.
Yes, and I agree for the user shell. It would still be nice to have a
minimalistic one for scripts, where extensions should be banned anyway.
Post by sin
xargs: Why?
I use it all the time, most frequently to call less or grep on a list of
files produced by some other command.

-Truls
Thorsten Glaser
2013-10-18 12:38:11 UTC
Permalink
Post by Truls Becken
bc dc
bc can be done on top of dc; BSD does that (the dc uses OpenSSL
for arbitrary-precision numbers).
Post by Truls Becken
killall
I object, killall should never, ever, be used. (Try it on a
Solaris system, for example.)
Post by Truls Becken
Post by sin
make: Do we really need this in sbase?
Make is a basic tool, useful in lots of situations.
I=E2=80=99d call it a development tool, normally.
Post by Truls Becken
It would be nice not to rely on GNU for this.
NetBSD make is portable (they use it for pkgsrc=C2=AE).
(MirBSD=E2=80=99s is, somewhat, but in a sucky way, and
I=E2=80=99ve not had time to redo it since I learned a
lot in the meantime; it was only a quick hack,
never planned to persist, anyway.)
Post by Truls Becken
Post by sin
sh: We agreed to do without a shell.
Yes, and I agree for the user shell. It would still be nice to have a
minimalistic one for scripts, where extensions should be banned anyway.
The problem is that all existing =E2=80=9Cminimalistic=E2=80=9D ones suck.
The posh approach (take pdksh and strip all extensions
away) is interesting, but posh is very buggy, and the
author said he=E2=80=99d base on mksh were he to restart now.
But maintainance effort is hell on this (and you *will*
accidentally remove things that are not extensions, and
you *will* run into the problem that POSIX is, slowly,
standardising most of them like $((=E2=80=A6)) and $'=E2=80=A6' now).

That being said, mksh/lksh makes a nice /bin/sh for most systems=E2=80=A6

bye,
//mirabilos
--=20
This space for rent.
Strake
2013-10-18 12:53:17 UTC
Permalink
Post by sin
find: Useless, just do `du -a | grep blabla'
Yes, this is what one does on Plan 9.
Post by sin
I realize when xargs is useful, I just hate it.
Yep, the basic function is sane, but the other crap, insert mode and
quotation and such, loses.
Post by sin
Post by sin
find: Useless, just do `du -a | grep blabla'
I'm not interested in disk usage, but finding files based on certain
properties, such as update time, ownership, permissions, etc.
du -a | cut -f 2

Cheers,
Strake
Truls Becken
2013-10-18 13:45:11 UTC
Permalink
Post by Strake
Post by Truls Becken
I'm not interested in disk usage, but finding files based on certain
properties, such as update time, ownership, permissions, etc.
du -a | cut -f 2
Yes, but how do you filter the result on some combination of file attributes?

du -a | cut -f2 | xargs ls -dl | grep <awful pattern> | sed 's/.*:[0-9]* //' ?

And then you would have a hard time specifying pattern for "newer than 5 days".

If you want to do it this way around, a command is missing for filtering a
list of file names, as already mentioned by Raphaël. It could be generalised
to a command for filtering stdin by calling a predicate command, e.g:

du -a | cut -f2 | filter test -w

For this, test is missing "newer than 5 days" since that's what find does.

-Truls
Truls Becken
2013-10-18 18:48:46 UTC
Permalink
Post by Truls Becken
If you want to do it this way around, a command is missing for filtering a
list of file names, as already mentioned by Raphaël. It could be generalised
du -a | cut -f2 | filter test -w
Just for kicks I wrote filter. With this, the above gives same result as:

find . -perm +222

At least after sorting both outputs, and as long as the fact that these
particular tests do not check the same thing does not bite you. Not sure if it
is useful. It is certainly not standard. As already mentioned, test does not
provide the same set of tests as find.

-Truls
Evan Gates
2013-10-18 20:24:28 UTC
Permalink
Post by Truls Becken
find . -perm +222
Is this not what stest does?

-emg
Evan Gates
2013-10-18 20:31:31 UTC
Permalink
Post by Truls Becken
du -a | cut -f2 | filter test -w
I also find it worth mentioning that newline delimited lists of
filenames are not safe as newlines are legal in filenames. This also
disregards the possibility of tabs in filenames (and many other
solutions run into problems with whitespace as well).

find -exec and shell globs are good solutions as they don't depend on
printing and then reading a list of files. find -print0 may be
acceptable, but is not portable. xargs runs into many of the same
problems. (If you want to be POSIX compatible and feel that you must
print and read paths, try find ././ and use "././" as a separator. But
man is that gross.)

I vaguely recall reading about a set of tools that someone was writing
(I think it was cls?) that solved many of these problems by using
environment variables RS and FS or similar (like awk) to format their
output. Seeing as sbase is not aiming for POSIX compliance perhaps
something along these lines is worth considering?

-emg
Alex Pilon
2013-10-18 14:41:55 UTC
Permalink
Post by Strake
Post by sin
find: Useless, just do `du -a | grep blabla'
Yes, this is what one does on Plan 9.
Which doesn't necessarily make it right.
Post by Strake
Post by sin
I realize when xargs is useful, I just hate it.
Yep, the basic function is sane, but the other crap, insert mode and
quotation and such, loses.
So just don't implement the insane bits! Does anybody here use anything
but ‘-d’ or ‘-0’ and/or ‘-L’?
Post by Strake
Post by sin
I'm not interested in disk usage, but finding files based on certain
properties, such as update time, ownership, permissions, etc.
du -a | cut -f 2
No. ‘du -0 | cut -f 2-’, at least.

And what the heck? That has nothing to do with what *you* just quoted above.

We're talking about checking *update time*, *ownership*, *permissions*,
etc., not the file name. This means I have to wrap in shell, and before
I know it, my little traversal of a few million files (not kidding, real
use case, though it's noticeable for smaller numbers) takes longer, not
to mention that it's more code.

du -a \
| cut -f 2- \
| while read -rd f; do
if (($(stat --format=%a "$f") & 0111)) &&
[ $(stat --format=%u "$f") = $(id -u) ] &&
[ "$f" -nt $reference ] &&
[ -f "$f" ]; then
chmod a-x "$f"
fi
done

Still have to figure out how to make it work with ‘du -a0’, without
using the following.

awk 'BEGIN { RS="\0" } { sub("^[0-9]+[[:space:]]*", "", $0); print $0 }'

I've heard that the above is non-portable due to the use of nul as the
record separator. Should we care? It neither works with nawk or Plan 9
awk.

You can cut a line or two if in the previous while loop if you make it
slightly less readable but the point still stands, whereas


find -print0 -perm /111 -uid $(id -u) -type f -cnewer $reference | xargs -0 rm

is short and sweet.

Please don't implement ‘-delete’. It's both redundant and not in POSIX.
Are there any other considerations that would make us want it?

du's output is standardized
(http://pubs.opengroup.org/onlinepubs/009604499/utilities/du.html#tag_04_40_10),
mercifully, though that doesn't mean it makes handling a few special
cases easy. Should we care about any such? The only ones that I can
think of right now are newlines in the file names, which is pretty
stupid but possible.

Using du is brittle but you can use it without fear the overwhelming
majority of the time. Still, if you advocate strong typing as a Haskell
programmer, why wouldn't you in shell (well, not shell itself) as
appropriate?

Regards,

Alex Pilon
Alex Pilon
2013-10-19 16:44:13 UTC
Permalink
Apologies for the spam, some typos here
 though you likely knew what I
meant.
Post by Alex Pilon
No. ‘du -0 | cut -f 2-’, at least.
`du -a 
`
Post by Alex Pilon
du -a \
| cut -f 2- \
| while read -rd f; do
`while read -rd '' f; do`

Still not portable outside of bash and mksh due to the use of `-d`, to
the detriment of busybox or d?ash users..

Regards,

Alex Pilon
Simon Lieb
2013-10-18 12:03:28 UTC
Permalink
Hi,
Post by sin
Hi all,
I've been thinking, besides the commands that are already listed in TODO
and all the unimplemented options for existing commands, what other commands
do we want to include in sbase?
May sbase be interested in a small utility called « slow » I wrote some
time ago to slow down down input to output ? I can make a patch for
sbase.

I mainly use it with ii and to avoid trigger anti‐flood system and some
other cases.

See my git repository[0].

Regards,


[0]: git:://home.tibu.fr/slow
--
Simon Lieb
Simon Lieb
2013-10-18 12:05:31 UTC
Permalink
Mmh, I should have attached an archive.

Regards,
--
Simon Lieb
Chris Down
2013-10-18 12:13:08 UTC
Permalink
Post by Simon Lieb
May sbase be interested in a small utility called « slow » I wrote some
time ago to slow down down input to output ? I can make a patch for
sbase.
I mainly use it with ii and to avoid trigger anti‐flood system and some
other cases.
In my opinion, this is too niche for sbase. Thanks for the code, though,
although I mostly use pv -L (and yes, pv sucks).
Simon Lieb
2013-10-18 19:53:28 UTC
Permalink
Post by Chris Down
Post by Simon Lieb
May sbase be interested in a small utility called « slow » I wrote some
time ago to slow down down input to output ? I can make a patch for
sbase.
I mainly use it with ii and to avoid trigger anti‐flood system and some
other cases.
In my opinion, this is too niche for sbase. Thanks for the code, though,
although I mostly use pv -L (and yes, pv sucks).
I understand. Didn't know about pv, thanks for the hint.
--
Simon Lieb
Louis Santillan
2013-10-24 03:50:56 UTC
Permalink
Late to the party, but in response to Truls email request on
10/18/2013, there's some less sucky lossless compression/decompression
code at <https://code.google.com/p/miniz/> (Zip/deflate & zlib
compatible, small, fast) and <https://code.google.com/p/lzham/>
(nearly LZMA compression with much faster decompression, Windows only
for the time being :/).

-Louis

Continue reading on narkive:
Loading...